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(57) New findings such as the highlight theory which 
determine the impression of a face are utilized for the 
technique of modifying a photographed facial image and 
the makeup technique. A method according to the 
present invention includes determining a desirable fece 
desired by a person who is subjected to facial modifica- 



tion or makeup, and performing image processing like 
shape-merging on an original facial image or an image 
of a fece of the subject person based on the desirable 
face to thereby prepare images of selectable modified 
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Description 

TECHNICAL FIELD 

[0001 ] The present invention relates to a technique 
of modifying facial images and a makeup art of putting 
makeup on a face using the mocfifying technique. 

BACKGROUND OF THE INVENTION 

[0002] The present inventor has been making stud- 
ies, for a long time, on factors which determine the faces 
of persons, particularly facial impressions, and beauty 
and ugliness, from the viewpoints of plastic surgery and 
cosmetic surgery. The study has provided several new 
findings. One of them is the "highlight theory". Conven- 
tionally, it was a general thought that the components of 
a face, such as eyebrows, eyes, a nose and lips, would 
become significant factors to determine a facial impres- 
sion. The research done by the present inventor how- 
ever shows that the prime factor to determine a facial 
impression is the shading state of a face, i.e., the states 
of a bright area and a dark area, particularly, the state of 
the brightest area or highlighted area. The shading 
state relates to shading on a face which occurs under 
illumination of the natural light and reflects or 
expresses the three-dimensional shape of the face. The 
states of a highlighted area in such shading particularly 
affects a facial impression. 

[0003] The analysis based on this highlight theory 
classifies faces into three general types. As exemplarily 
shown in Figure 5. the shapes of highlighted areas are 
an inverted triangle type [(a) in Figure 5], an oval type 
[(b) in Figure 5] and a clover-leaf type [(c) in Figure 5]. A 
typical beautiful face in each facial type has a good 
shape of the associated highlighted area. With regard to 
an inverted triangle type, for example, no or little distor- 
tion or deformation on its shape is a significant factor in 
giving a good impression on a face. 
[0004] Another finding is a theory on the shapes of 
faces people feel beautiful or preferable. Conventional 
theories on facial beauty include canon theory or a the- 
ory based on golden cut, and the average theory which 
has recently come up. The average theory means a the- 
ory that the face that appears most beautiful among 
those of a certain human race is an average face of the 
individual faces of that race. Faces based on those the- 
ories merely provide general "beautiful faces". In other 
words, those theories cannot give ideal post-makeup 
faces which are makeup aims applicable to makeup 
methods. Unlike those theories, the theory found by the 
present inventor can be put to a practical use in makeup 
methods. 

[0005] According to what has been newly found by 
the present inventor, through the following procedures, 
rt is possible in most occasions to find the most beautiful 
or preferable face for each specific person, a the most 
beautiful or preferable face that can be made by 



makeup available to that individual. The fundamental 
theory is such that if faces are produced by combining a 
face of given standards, specifically, a face generally 
considered beautiful or a face preferred by an individual 

5 and the actual face of that particular individual at the 
proper mixing ratios through an image processing tech- 
nique, an "ideal face" which is considered most beautiful 
or preferable by that individual can be found in most 
cases from the foregoing mixed faces. Specifically, one 

10 of model faces, which are. for example, faces of pre- 
ferred talents or actresses or actors, or "beautiful faces" 
originated from the average theory and the like, is 
selected as a desirable face. Based on this desirable 
face, the actual face of the person is subjected to image 

is processing like morphing so that the actual face is 
mixed with the desirable face to approach the latter. 
This process provides a plurality of "modified faces" with 
different mixing ratios or similarity levels with respect to 
the desirable face. In most cases, those "modified - 

20 faces" contain an "ideal face" that person considers 
most beautiful or preferable. The data obtained by the 
present inventor shows that persons in 48 cases out of 
50 could find "ideal faces" they thought most beautiful or 
preferable. In the other two cases, "ideal faces" could be 

25 found by modifying the desirable faces. The mixing ratio 
in every case ranged from 10 to 50%. 
[0006] It is to be noted that one model face itself 
should not necessarily be the most preferable face for a 
particular person. That is, the "ideal face" a particular 

30 person can find for herself or himself is a "mocfif ied face" 
which is resulted from the mixing of a model face with 
her or his own face, not the model face itself The actual 
results of the experiments conducted by the present 
inventor show that most subject persons did not prefer 

35 model faces using "beautiful faces" created by the aver- 
age theory or the like. 

[0007] The present invention has been devised to 
effectively utilize the above-described findings and aims 
at using those findings to a technique of modifying, for 
40 example, photographed facial images and to a makeup 
technique. 

DISCLOSURE OF THE INVENTION 

45 [0008] According to the present invention, facial 
images picked up by a still camera, a video camera or 
the like are modified as follows. First a desirable face is 
determined, and then an original facial image is sub- 
jected to image processing to become closer to this 

so desirable face so that a modified facial image is 
acquired. 

[0009] Model faces can be used in determining a 
desirable face The faces of preferable talents or 
actresses or actors can be used as model faces. If one 
55 does not have any preferable talent or the like, standard 
good-looking faces acquired from a plurality of beautiful 
faces which have previously been prepared correspond- 
ing to human races or like according to the foregoing 
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average theory may be used, tn the foregoing method, It 
may aiso be arranged that an original facial image is 
subjected to image processing to become closer to a 
desirable face, thereby forming images of a plurality of 
selectable modified faces with different similarity levels 5 
with respect to the desirable face, and then a selection 
is made from those selectable modified faces to acquire 
a modified facial image. This further facilitates acquiring 
a face ideal to the subject person. 
(001 0] It is preferable to use the foregoing highlight 10 
theory in modifying and making an original facial image 
closer to a desirable face. That is, the key point is to 
make the states of highlighted areas of a facial image to 
be modif ied closer to those of a desirable face. This can 
provide a more beautiful and preferable face and thus is 
facilitate approaching to a desirable face. 
[0011] This facial image modifying method is an 
application of the foregoing theory on the shapes of 
faces that people think beautiful or preferable. Unlike 
the conventional photograph modification techniques 20 
which mainly depend on a manual work, this method 
can automate the modification work by electronic 
processing. What is more, modification results can give 
greater satisfaction. This method of the present inven- 
tion can be adapted to operational simulation in plastic 2s 
surgery and cosmetic surgery as well as ordinary pho- 
tograph modification. 

[0012] In a makeup simulation method embodying 
the present invention and a makeup method which uses 
this simulation method, a desirable face is determined so 
first. In the case where a makeup instructor instructs a 
subject person how to put makeup, for example, a desir- 
able face may be determined by asking the subject per- 
son about preferences. Alternatively, model faces may 
be used in determining a desirable face. The faces of 35 
talents, or actresses or actors the subject person like 
can be used as model faces. If the subject person has 
no preference for any talent or the like, averages of 
good-looking faces which have previously been pre- 
pared corresponding to human races or the like may be *o 
used as model faces. In the former case, a preferred 
desirable face can be selected from model faces using 
the name or the like of the preferable talent or the like as 
an index. In the latter case, a preferred desirable face 
should be selected from model faces of standard beauty <s 
which has previously been input as data. In both cases 
a selection is made while sequentially displaying the 
model faces on a monitor screen or based on the type 
of the face of the subject person. In the latter case, the 
model faces should not necessarily be shown to the so 
subject person. 

[0013] Once the desirable face is decided, then 
images of imagined post-makeup faces based on the 
desirable face or post-makeup faces which are imag- 
ined as preferable are prepared. This involves a process ss 
of mixing the face of the subject person with the desira- 
ble face to make the former face doser to the latter by 
performing image processing like shape-merging on the 



image of the face of the subject person. Then, an ideal 
post-makeup face the subject person desires is deter- 
mined from the imagined post-makeup faces. Specifi- 
cally, a plurality of imagined post-makeup faces with 
different mixing ratios or similarity levels with respect to 
the desirable face are acquired through the foregoing 
image processing, and a preferable selection is made 
from those faces as an ideal post-makeup face within 
the range of the similarity levels whlchian.be obtained 
by makeup. Thus, the ideal post-makeup face which is 
expected to be finally obtained is given in advance. That 
is, the subject person can know the makeup-finished 
face in a short time. As apparent from the above, the 
significant feature of the makeup method embodying 
the present invention lies in that a final, makeup-finished 
face can be presented in a short time by determining a 
desirable face on which a makeup procedure is based 
and acquiring an ideal post-makeup face with respect to 
this desirable face. In this feature is effectively used the 
above-descrrbed theory on the shapes of faces people 
think beautiful or preferable. 

[001 4] When an ideal post-makeup face is decided 
through the foregoing makeup simulation, a makeup 
procedure is derived from this ideal post-makeup face. 
Specifically, a sequence of makeup procedures for 
achieving the ideal post-makeup face, such as selecting 
which portions of eyebrows to be shaved and where to 
draw eyebrows, selecting the lines and ranges for eye 
liners and eye shadows, selecting the colors of eye 
shadows, setting the drawing range~for~a lipstick -and 
separately applying Afferent foundations, is derived 
from a previously set makeup program. Based on this 
makeup procedure, makeup is applied to the subject 
person. This makes it possible to accurately realize the 
ideal post-makeup face or the makeup state the subject 
person has previously found appropriate, on the face of 
the subject person. That is, makeup which the subject 
person desires can be done freely and in a short time. 
[0015] To realize the foregoing feature of the 
makeup method of the present invention which lies in 
the preparation of an ideal post-makeup face through 
image processing based on a desirable face or a model 
face, it is important to mix the current face of a subject 
person with a model face through image processing to 
get the former face closer to the latter. That is, it is 
essential to acquire an ideal post-makeup face by mak- 
ing the current face of the subject person closer to the 
reference model face by image processing. The feature 
also requires an effective scheme or standards tor mak- 
ing the current face of the subject person doser to the 
ideal post-makeup face by actual makeup. This can be 
accomplished by the foregoing highlight theory. 
[001 6] Based on the highlight theory which teaches 
highlighted areas considerably affect the impression of 
a face and a beautiful and preferable face can be made 
by erasing any unnecessary shade (dark area) to make 
the shapes of the highlighted areas better, it becomes 
relatively easy to get the face of a subject person closer 
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to a model face through image processing and it is also 
easy to make the face of the subject person closer to a 
desirable face by implementing such a makeup as to 
adjust the states of the highlighted areas in association 
with those of the model face (desirable face). In other 5 
words, a more effective scheme of making the face of 
the subject person closer to the reference model face 
can be acquired based on the highlight theory. 
[001 7J The image processing for preparing the 
images of imagined post-makeup faces on the basis of 10 
the highlight theory involves a process of making the 
shapes of the eyebrows, the eye lines, the lip line, and 
so forth closer to those of a desirable face within a range 
where modification by makeup is possible, in addition to 
the basic process of getting the states of highlighted is 
areas on a face closer to those of the desirable face 
within a range where modification by makeup is possi- 
ble. A shape-merging scheme or the like can be also 
used in this case. 

[0018] Based on the hig Wight theory, attention 20 
should be paid to a greatest highlighted area as an 
impressing element of a face in the makeup simulation 
method and makeup method, too Since makeup about 
highlighted areas is accomplished by separately apply- 
ing cfifferent foundations, the types of foundations and 25 
procedures for applying them separately are derived as 
essential factors from an ideal post-makeup face. 
[0019] To accomplish the process, image process- 
ing is performed to divide the ideal post-makeup face 
into a plurality of areas originating from differences 30 
between bright and dark areas like a pattern of contour 
lines. Then, color data is acquired for each segmented 
area. Color data in this case consists of hue and bright- 
ness, and chroma if necessary, and R, G and B which 
are used in a television system or C. M. Y and K which 35 
are basic printing colors in a computer system are nor- 
mally used. Based on the color data of the individual 
areas and a previously set foundation program, founda- 
tions to be applied on the individual areas are deter- 
mined. Finally, the thud determined foundations are 40 
applied on the face of the subject person in accordance 
with the segmented areas. 

[0020] One makeup simulation method according 
to the present invention and a makeup method using 
this simulation method therefore include the following 45 
steps: (a) determining a desirable face and performing 
image processing on an image of a face of a subject 
person based on the desirable face, thereby forming 
images of imagined post-makeup faces; (b) displaying 
the imagined post-makeup faces, prepared in the step so 

(a) , on a monitor screen or the like and selectively deter- 
mining an ideal post-makeup face desired by the sub- 
ject person from the displayed imagined post-makeup 
faces; (c) deriving a makeup procedure from an image 

of the ideal post-makeup face, determined in the step ss 

(b) , based on a previously set makeup program; and (d) 
applying makeup on the face of the subject person in 
accordance with the makeup procedure derived in the 



step(c). 

[0021] Another makeup simulation method embod- 
ying the present invention and another makeup method 
using this simulation method, particularly, a makeup 
simulation method for applying makeup based on dark 
and bright states on a face and a makeup method using 
this simulation method, include the following steps: (e) 
determining a desirable face and performing image 
processing on an image of a face of a subject person 
based on the desirable face, thereby forming images of 
imagined post-makeup faces; (f) displaying the imag- 
ined post-makeup faces, prepared in the step (e), on a 
monitor screen or the like and selectively determining 
an ideal post-makeup face desired by the subject per- 
son from the displayed imagined post-makeup faces; 
(g) performing image processing to segment an image 
of the ideal post-makeup face, determined in the step 
(f), to a plurality of areas in regard to bright areas and 
dark areas; (h) acquiring color data for the segmented 
areas obtained in the step (g); (i) determining founda- 
tions to be applied on individual areas based on the 
color data for those areas, acquired in the step (h), and 
a previously set foundation program; and (j) separately 
applying foundations, determined in the step (i). on the 
face of the subject person in association with the indi- 
vidual areas. 

[0022] The makeup method based on dark and 
bright states may be modified in such a way as to be 
able to print the image of the ideal post-makeup face, 
segmented into a plurality of areas in regard to bright 
areas and dark areas in the step (g), in an actual size of 
the face of the subject person, to cut out individual 
areas from the printed image to prepare a pattern mask 
for foundation, and to execute separate application of 
foundations in the step (k) by using this pattern mask. 
This design can allow different foundations to be sepa- 
rately applied on the associated areas easily and accu- 
rately. That is, separate application of different 
foundations associated with an ideal post-makeup face 
can be carried out easily and accurately. 
[0023] The makeup method which is based on the 
highlight theory may be modified in such a manner as to 
be able to project the image of the ideal post-makeup 
face, segmented into a plurality of areas in regard to 
bright areas and dark areas in the step (g), in an actual 
size on the face of the subject person, and to execute 
separate application of foundations in the step Q) based 
on this projected image. This modification can also per- 
mit different foundations to be separately applied on the 
associated areas easily and accurately. 
[0024] Another makeup method according to the 
present invention which is based on dark and bright 
states includes the following steps: (A) determining a 
desirable face and performing image processing on an 
image of a face of a subject person based on the desir- 
able face, thereby forming images of imagined post- 
makeup faces; (B) displaying the imagined post- 
makeup faces, prepared in the step (A), on a monitor 
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screen or the like and selectively determining from the 
displayed imagined post-makeup faces an ideal post- 
makeup face the subject person desires; (C) performing 
image processing to segment an image of the ideal 
post-makeup face, determined in the step (6), to a plu- 5 
rality of areas in regard to bright areas and dark areas; 
(D) acquiring color data for the segmented areas 
obtained in the step (C); (E) determining foundations to 
be applied on individual areas based on the color data 
for those areas, acquired in the step (D), and a previ- 10 
ously set foundation program; (F) printing foundations, 
determined in the step (E) in association with the plural- 
ity of areas segmented in regard to bright areas and 
dark areas in the image processing in the step (C), on a 
thin base material to prepare a transfer film; and (Q) is 
transferring foundations, printed on the transfer film pre- 
pared in the step (F), on the face of the subject person 
to thereby apply foundations, associated with the indi- 
vidual areas, on the face of the subject person. 
[0025] A further makeup method according to the 20 
present invention which is based on dark and bright 
states includes the following steps: (H) determining a 
desirable face and performing image processing on an 
image of a face of a subject person based on the desir- 
able face, thereby forming images of imagined post- 25 
makeup faces; (I) displaying the imagined post-makeup 
faces, prepared in the step (H), on a monitor screen or 
the like and selectively determining from the displayed 
imagined post-makeup faces an ideal post-makeup face 
the subject person desires; (K) printing an image of the so 
ideal post-makeup face in the bright and dark states, 
determined in the step (I), on a thin base material using 
foundations to prepare a foundation transfer film; and 
(L) transferring foundations, printed on the foundation 
transfer film prepared in the step (K), on the face of the 35 
subject person to thereby apply the foundations on the 
face of the subject person. 

[0026] Those methods can repeatedly realize 
makeup tor an ideal post-makeup face by carrying out 
simple procedures, such as transferring foundations 40 
from a transfer film accurately corresponding to founda- 
tions for the ideal post-makeup face and properly shad- 
ing off the transferred foundations at the boundaries 
between the individual areas. Thus, desired makeup 
can be carried out more easily and in a shorter time. *5 
[0027] A makeup support apparatus according to 
the present invention for use in the foregoing makeup 
simulation methods and makeup methods comprises 
image storage means tor storing images of a plurality of 
model faces; imaging and input means for imaging and so 
inputting a face of a subject person; a monitor for dis- 
playing a necessary image; and processing means 
capable of performing a process of causing an image of 
the face of the subject person to approach a desirable 
face already determined to thereby form images of ss 
imagined post-makeup faces, a process of displaying 
the imagined post-makeup faces on a screen of the 
monitor, and a process of deriving a makeup procedure 



from a determined ideal post-makeup face. 
[0028] It is preferable that the processing means in 
the makeup support apparatus may further perform a 
process of segmenting the determined ideal post- 
makeup face to a plurality of areas in regard to bright 
areas and dark areas, a process of acquiring color data 
for the segmented areas, and a process of determining 
foundations to be applied on individual areas based on 
the acquired color data for those areas. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0029] 

Figure 1 is a diagram for explaining one example of 
image processing in a facial image modifying 
method. 

Figure 2 is a structural diagram of a makeup sup- 
port apparatus according to one carrying-out mode 
of the present invention. 

Figure 3 is an explanatory diagram concerning a 
highlight theory. 

Figure 4 is a diagram for explaining segmentation 
into highlighted areas. 

Figure 5 is an explanatory diagram of the types of 
highlighted areas. 

BEST MODE FOR CARRYING OUT THE INVENTION 

[0030] Hereinbelow, carrying-out modes of the 
present invention will described. To begin with, a carry- 
ing-out mode of a facial image modifying method 
according to the present invention will be described. 
Normally, a system which includes a data processing 
apparatus like a personal computer and a monitor con- 
nected to the apparatus is used in implementing the 
facial image modifying method of the present invention. 
An image of a face to be modified or an original facial 
image is put into the data processing apparatus. Then, 
a desirable face is selected from previously prepared 
model faces, e.g., multiple standard faces which have 
been prepared based on the average theory, and the 
original facial image is subjected to image processing 
based on this desirable face. 
[0031 ] The image processing is executed by a pro- 
gram which employs an image processing scheme like 
morphing or twining. Specifically, the coordinate values 
of facial constituents, such as eyebrows, eyes, a nose 
and a mouth, of both an original facial image M and a 
desirable face T, and the coordinate values of bright and 
dark patterns of each face are acquired as shown in Fig- 
ure 1, for example, and the original facial image M is 
mixed with and transformed toward the desirable face T 
based on those coordinate values. 
[0032] More specifically, first essential portions, 
such as eyebrows, eyes, a nose, a mouth and the lines 
of highlighted areas, are plotted for each of the original 
facial image M and the desirable face T. For example, a 
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tip portion of the nose is expressed as "0". the eye lines 
as "1 -2" and "3-4". and the eyebrows as "5-6" and "7-8". 
Then, those points on the original facial image M are 
shifted toward the corresponding points on the desirable 
face T by a morphing program. The lines of the high- 5 
lighted areas or the like are shifted or their inclinations 
are changed by a twining program. Such image 
processing by shifting points or the like includes a plu- 
rality of phases according to the degree of such shifting. 
That is, selectable modified faces (imagined post- 10 
makeup faces in the case of a makeup method which 
will be described later) corresponding to those phases 
can be obtained. If point shifting is performed every one 
tenth of the distance between corresponding points on 
both facial images, for example, there are ten phases is 
and there are thus ten imagined post-makeup faces. 
According to the experiments conducted by the present 
inventor, most of the times, subject persons found their 
"ideal faces" (selectable modified faces or imagined 
post-makeup faces) on selectable modified faces at the 20 
second or third phase, i.e., selectable modified faces 
which had approached to their desirable faces by 
degrees of 20 to 30%. That is, while there is a desirable 
face as an image, a face too dose to this desirable face 
comes off the actual "ideal face". 25 
[0033] As the above process forms the images of a 
plurality of selectable modified faces with different simi- 
larity levels with respect to a desirable face, they are dis- 
played on the screen of the monitor to permit the subject 
person to make the best selection, which is treated as a 30 
modified facial image. 

[0034] Now, carrying-out modes of a makeup simu- 
lation method and a makeup method according to the 
present invention will be described. The makeup 
method of the present invention is carried out by using a 35 
makeup support apparatus. The makeup support appa- 
ratus of which structure is illustrated in Figure 2 includes 
a data processing apparatus 1 , a storage device 2, a 
monitor 3 and an image sensing unit 4. The data 
processing apparatus 1, which executes various kinds 40 
of data processing required in the makeup method of 
the present invention, is a personal computer or the like. 
The storage device 2 is used to store data of model 
faces which are used in the makeup method of the 
present invention. An internal memory device or an 45 
external memory device is used as the storage device 
2. 

[0035] In an exemplified case where a makeup 
instructor instructs a subject person a makeup proce- 
dure using such a makeup support system, the makeup so 
method proceeds as follows. First, the instructor asks 
the subject person specific questions to determine the 
impression of a face desired by the subject person or a 
desirable face. Model faces are used to make a desira- 
ble face concrete. As model feces are used image data ss 
of faces of popular talents, actresses or actors or the 
like, or standard good-looking faces prepared in associ- 
ation with human races or the like, which are stored in 



the storage device 2. A desirable face is selected from 
the model faces by using an average face for each race, 
each era, each personality, each work or the like or the 
name of a specific talent, or is selected from various 
model faces sequentially displayed on the screen of the 
monitor 3. 

[0036] Once the desirable face is determined, the 
face of the subject person is picked up by the image 
sensing unit 4 and its image is sent into the storage 
device 2. By performing image processing on the image 
of the face of the subject person using the data process- 
ing apparatus 1, the image of the face of the subject 
person and the image of the desirable face are mixed to 
make the former image closer to the latter. In the image 
processing, the original facial image of the face of the 
subject person acquired in the above process is sub- 
jected to the same scheme as employed in the above- 
described modifying method to make the highlighted 
areas, the shapes of the eyebrows, the eye lines, the lip 
line and the like on the face approach those of the desir- 
able face step by step. 

[0037] Figure 3 exemplifies highlighted areas, (a) in 
Figure 3 shows the states of bright and dark areas on 
the original facial image of the face of the subject per- 
son, and (b) in Figure 3 shows the states of bright and 
dark areas made close to those of the desirable face in 
image processing. In the figure, the densely dotted por- 
tions represent relatively dark portions or dark areas 
and a portion surrounded by the dark portions represent 
the brightest area (highlighted area). As apparent from 
this example in which the highlighted area belongs to an 
oval shape, the current face of the subject person is off 
the ideal oval shape and the face of the subject person 
can be made closer to the desirable face which has an 
ideal oval highlighted area by mainly modifying the off 
portions. 

[0038] As this process proceeds, the images of a 
plurality of imagined post-makeup faces with different 
similarity levels with respect to a desirable face are 
formed as described above, and those images are dis- 
played on the screen of the monitor 3 to permit the sub- 
ject person to make a selection. If the subject person 
agrees with any of the displayed images, the selected 
face is set as an ideal post-makeup face. 
[0039] Once the ideal post-makeup face is deter- 
mined, a sequence of makeup procedures about por- 
tions of eyebrows to be shaved and where to draw 
eyebrows, the lines and ranges for eye liners and eye 
shadows, the drawing range for a lipstick and separate 
application of different foundations, is derived from the 
ideal post-makeup face. This sequence is carried out by 
a previously set makeup program. 
[0040] With regard to foundation, in particular, how 
to apply foundations is derived by performing the follow- 
ing processes under the control of a foundation program 
which is included in the makeup program. First image 
processing is performed to divide the ideal post- 
makeup-face into a plurality of bright and dark areas, 
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e.g.. two to four areas. This process is exemplified in 
Figure 4. tn the exemplified diagram, the face is seg- 
mented to four areas with different brightnesses (black- 
filled portion, densely-dotted portion, thirty-dotted por- 
tion and white portion). Next, color data for the seg- 5 
merited areas are acquired. The color data in this case 
are based on R, G and B used in a television system. 
Based on the obtained color data of each area, a foun- 
dation to be applied on that area is determined. 
[0041] As the makeup procedure needed for mak- 10 
ing up the face of the subject person into the ideal post- 
makeup face and cosmetics to be used in the makeup 
procedure are determined in the above manner, actual Claims 
makeup is finally applied to the face of the subject per- 
son based on what has been determined. In utilizing the 
thus derived makeup procedure and the like in the 
actual makeup, data about the makeup procedure, etc. 
should be displayed on the monitor screen or printed 
out 

[0042] According to a second carrying-out mode of 
the present invention, a pattern mask is used to apply 2. 
foundations on the face of a subject person. A pattern 
mask is formed (ike a pattern sheet by cutting out indi- 
vidual areas from a printed image of the face, seg- 
mented into a plurality of areas in regard to bright areas 2s 
and dark areas in the above-described image process- 
ing, in the real size of the face of the subject person. 
Then, the pattern mask is placed on the face of the sub- 
ject person and foundations for the individual areas are _ 
applied overthe pattern mask. ~ : : •>vr:,r&'__, 
[0043] A third carrying-out mode of the present 
invention employs a projection scheme to apply founda- 
tions on the face of a subject person. Specifically, the 
image of the face, segmented into a plurality of areas in 
regard to bright areas and dark areas in the above- 35 
described image processing, is projected on the face of 
the subject person in real size, and foundations for the 
associated areas are separately applied based on this 
projected image. 

[0044] A fourth carrying-out mode of the present 40 3. 
invention uses a transfer scheme to apply foundations 
on the face of a subject person. Specifically, first, in 
association with a plurality of areas segmented in 
regard to bright areas and dark areas in the above-dis- 
cussed image processing, foundations for the respec- 45 
tive areas are printed in an actual size on a thin base 
material to prepare a foundation transfer film. Alterna- 
tively, images of bright and dark states on the ideal post- 
makeup face determined in the above-described man- 
ner are printed on a thin base material using founda- so 
tions to prepare a foundation transfer film. Then, such a 
foundation transfer film is pressed against the face of 
the subject person to transfer the foundations. Finally, 
the proper shade-off process or the like is performed on 
the transferred foundations at the boundaries between 55 
the individual areas. The foundation transfer film in this 
case does not necessarily correspond to the entire face, 
but a partial transfer f flm for the upper eyelids or a por- 



tion around the nose, for example, may be used. 
[0045] According to the fourth carrying-out mode of 
the present invention, application of foundations to the 
face of the subject person is accomplished by directly 
printing the foundations on the face by means of a 
three-dimensional printer or the like. Specifically, an 
image with separately applied foundations, which is 
formed by the above-described image processing, is 
directly spray-printed on the face by * three-dimen- 
sional printer or the like which uses the necessary foun- 
dations as inks. 



A facial image modifying method comprising the 
steps of determining a desirable face, and causing 
an original facial image to approach an image of 
said desirable face through image processing to 
acquire a modified facial image. 

A makeup simulation method comprising the steps 
of: 

(a) determining a desirable face and perform- 
ing image processing on an image of a face of 
a subject person based on said desirable face, 
thereby forming images of imagined post- 
makeup faces; 

(b) displaying said imagined post-makeup 
faces, prepared in said step (a), on a monitor 
screen or the like and selectively determining 
from said displayed imagined post-makeup 
faces an ideal post-makeup face said subject 
person desires; and 

(c) deriving a makeup procedure from an 
image of said ideal post-makeup face, deter- 
mined in said step (b), based on a previously 
set makeup program. 

A makeup simulation method for executing makeup 
simulation based on shading states of a face, com- 
prising the steps of: 

(e) determining a desirable face and perform- 
ing image processing on an image of a face of 
a subject person based on said desirable face, 
thereby forming images of imagined post- 
makeup faces; 

(f) displaying said imagined post-makeup 
faces, prepared in said step (e), on a monitor 
screen or the like and selectively determining 
from said displayed imagined post-makeup 
faces an ideal post-makeup face said subject 
person desires; 

(g) performing image processing to segment 
an image of said ideal post-makeup face, 
determined in said step (f), to a plurality of 
areas in regard to bright areas and dark areas; 
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(h) acquiring color data for said segmented 
areas obtained in said step (g); and 
(t) determining foundations to be applied on 
individual areas based on said color data for 7. 
those areas, acquired in said step (h), and a s 
previously set foundation program. 

4. The makeup simulation method according to claim 
2 or 3, further comprising a step of preparing a plu- 
rality of model faces and determining said desirable io 
face based on said model faces. 

5. A makeup method comprising the steps of: 

(a) determining a desirable face and perform- is 
ing image processing on an image of a face of 

a subject person based on said desirable face, 
thereby forming images of imagined post- 
makeup faces; 8. 

(b) displaying said imagined post-makeup 20 
faces, prepared in said step (a), on a monitor 
screen or the like and selectively determining 
from said displayed imagined post-makeup 
faces an ideal post-makeup face said subject 
person desires; 25 

(c) deriving a makeup procedure from an 
image of said ideal post-makeup face, deter- 
mined in said step (b), based on a previously 
set makeup program; and 

(d) applying makeup on said face of said sub- so 
ject person in accordance with said makeup 
procedure derived in said step (c). 9. 

6. A makeup method for applying makeup based on 
shading states of a face, comprising the steps of: 35 

(e) determining a desirable face and perform- 
ing image processing on an image of a face of 
a subject person based on said desirable face, 
thereby forming images of imagined post- 40 
makeup faces; 

(f) displaying said imagined post-makeup 
faces, prepared in said step (e), on a monitor 
screen or the like and selectively determining 
from said displayed imagined post-makeup 45 
faces an ideal post-makeup face said subject 
person desires; 

(g) performing image processing to segment 
an image of said ideal post-makeup face, 
determined in said step (f), to a plurality of so 
areas in regard to bright areas and dark areas; 

(h) acquiring color data for said segmented 
areas obtained in said step (g) ; 

(i) determining foundations to be applied on 
individual areas based on said color data for ss 
those areas, aoquired in said step (h), and a 
previously set foundation program; and 

(j) separately applying foundations, determined 



in said step (i). on said face of said subject per- 
son in association with said individual areas. 

The makeup method according to claim 6, further 
comprising: 

(k) printing said image of said ideal post- 
makeup face, segmented into a plurality of 
areas in regard to bright areas and dark areas 
in said step (g) in claim 3. in an actual size of 
said face of said subject person; 
(I) cutting individual areas from a printed image 
obtained in said step (k) to prepare a pattern 
mask for foundation; and 
(m) executing separate application of founda- 
tions in said step Q) in claim 6 using said pat- 
tern mask prepared in said step (I). 

The makeup method according to claim 6. further 
comprising: 

(n) projecting said image of said ideal post- 
makeup face, segmented into a plurality of 
areas in regard to bright areas and dark areas 
in the image processing in said step (g) in claim 
3. in an actual size on said face of said subject 
person; and 

(0) executing separate application of founda- 
tions in said step (j) in claim 3 based on said 
image projected in said step (n). 

A makeup method for applying makeup based on 
bright and dark states of a face, comprising the 
steps of: 

(A) determining a desirable face and perform- 
ing image processing on an image of a face of 
a subject person based on said desirable face, 
thereby forming images of imagined post- 
makeup faces; 

(B) displaying said imagined post-makeup 
faces, prepared in said step (A), on a monitor 
screen or the like and selectively determining 
from said displayed imagined post-makeup 
faces an ideal post-makeup face said subject 
person desires; 

(C) performing image processing to segment 
an image of said ideal post-makeup face, 
determined In said step (B). to a plurality of 
areas in regard to bright areas and dark areas; 

(D) acquiring color data for said segmented 
areas obtained in said step (C); 

(E) determining foundations to be applied on 
individual areas based on said color data for 
those areas, acquired in said step (D), and a 
previously set foundation program; 

(F) printing foundations, determined in said 
step (E) in association with said plurality of 
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areas segmented in regard to bright areas and 
dark areas in said image processing in said 
step (C). on a thin base material to prepare a 
foundation transfer film; and 

(G) transferring foundations, printed on said s 
foundation transfer film prepared in said step 
(F), on said face of said subject person to 
thereby apply foundations, associated with said 
individual areas, on said face of said subject 
person. 10 

1 0. A makeup method comprising the steps of: 

(H) determining a desirable face and perform- 
ing image processing on an image of a face of is 
a subject person based on said desirable face, 
thereby forming images of imagined post- 
makeup faces; 

(I) displaying said imagined post-makeup 
faces, prepared in said step (H), on a monitor 20 
screen or the like and selectively determining 

an ideal post-makeup lace desired by said sub- 
ject person from said displayed imagined post- 
makeup faces; 

(K) printing an image of said ideal post-makeup 2s 
face, determined in said step (I), on a thin base 
material using foundations to prepare a foun- 
dation transfer fflm; and 
(L) transfening foundations, printed on said 
foundation transfer film prepared in said step 30 
(K), on said face of said subject person to 
thereby apply said foundations on said face of 
said subject person. 



a determined ideal post-makeup face. 

1 3. The makeup support apparatus according to claim 
12, wherein said processing means further per- 
forms a process of segmenting said determined 
ideal post-makeup face to a plurality of areas in 
regard to bright areas and dark areas, a process of 
acquiring color data for said segmented areas, and 
a process of determining foundations to be applied 
on individual areas based on said acquired cdor 
data for those areas. 

14. A foundation transfer film for use in separate appli- 
cation of foundations, said foundation transfer film 
having foundations printed on a thin base material 
in regard to bright areas and dark areas on a face. 



11. TTie makeup method according to any one of claims 35 
5 to 10. further comprising a step of preparing a 
plurality of model faces and determining said desir- 
able face based on said model faces. 

12. A makeup support apparatus for use in the makeup <o 
simulation method as recited in any one of claims 2 

to 4 or in the makeup method as recited in any one 
of claims 5 to 1 1 , comprising: 

image storage means for storing images of a 45 
plurality of model faces; 
image pick-up and input means for imaging and 
inputting a face of a subject person; 
a monitor for displaying a necessary image; 
and so 
processing means capable of performing a 
process of causing an image of said face of 
said subject person to approach a desirable 
face already determined to thereby form 
images of imagined post-makeup faces, a ss 
process of displaying said imagined post- 
makeup faces on a screen of said monitor, and 
a process of deriving a makeup procedure from 
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Description 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention s 

[0001 ] The present invention relates to a three-dimen- 
sional beauty simulation client-server system to carry 
out beauty simulations based on a user's face model 
data. 10 

2. Description of the Related Art 

[0002] The invention described in Japanese Patent 
Laid-Open No. H6-319613, for example, comprises a 1$ 
conventional beauty simulation apparatus. This inven- 
tion discloses a face makeup support apparatus using 
which makeup may be applied to a displayed face by 
simulating lipstick application, face powdering and eye- 
brow shaping on the image of a face displayed in an 20 
image display apparatus. 

[0003] The conventional beauty simulation apparatus 
entails the problem that it can only carry out flat image 
processing, and does not appear realistic. Furthermore, 
it cannot be used over a computer network. 25 

SUMMARY OF THE INVENTION 

[0004] The present invention was created in order to 
resolve these problems, and an object thereof is to pro- 30 
vide a three-dimensional beauty simulation client-server 
system that can display a user's face in a three-dimen- 
sional fashion and provide a more realistic beauty sim- 
ulation. 

[0005] The three-dimensional beauty simulation cli- 35 
ent-server system pertaining to the present invention in- 
cludes a shop-based client that obtains and transmits 
three-dimensional user data, a makeup simulation unit 
that receives and stores the three-dimensional shape 
data from the shop-based client and carries out makeup *o 
simulation based on the three-dimensional shape data 
in response to requests from the user, and a server that 
includes a data control unit that analyzes the user's op- 
eration record and generates administrative informa- 
tion. 45 
[0006] It is preferred that the shop-based client in- 
clude a plurality of cameras by which to obtain images 
of the user from a plurality of viewpoints, a correspond- 
ing point search unit that receives each item of image 
data obtained from the plurality of cameras, analyzes so 
the plurality of images, and searches for corresponding 
points that match each other, a three-dimensional shape 
recognition unit that analyzes the searched correspond- 
ing points and recognizes the three-dimensional shape 
of the object, a geometric calculation unit that sets a line ss 
of sight based on the recognition results from the three- 
dimensional shape recognition unit and generates an 
image from the prescribed line of sight through geomet- 



ric conversion of the data based on the set tine of sight, 
a display unit that displays the image generated by the 
geometric calculation unit, and a communication means 
that transmits the image data generated by the geomet- 
ric calculation unit to the server. 
[0007] It is further preferred that the makeup simula- 
tion unit of the server include a receiving unit that re- 
ceives the three-dimensional shape data, a database 
that stores the received three-dimensional shape data, 
and a makeup simulation providing unit that provides a 
makeup simulation in response to requests for such sim- 
ulation, and that the data control unit of the server in- 
clude a user information analyzer that receives the op- 
eration history of the user and analyzes the trends there- 
in, a control database that stores the analyzed data, an 
information processing unit that reads out data from the 
control database in response to external requests and 
processes the data in accordance with the requests, and 
a transmitting/receiving unit that transmits the output of 
the information processing unit to the requesting source 
and receives requests from the requesting source. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0008] 

Fig. 1 is a drawing showing the overall system per- 
taining to an embodiment of the present invention; 

Fig. 2 is a drawing showing the basic construction 
of the server pertaining to the embodiment of the 
present invention; 

Fig. 3 is a drawing showing the basic construction 
of the shop-based client pertaining to the embodi- 
ment of the present invention; 

Fig. 4 is an example of the display screen of the 
shop-based client pertaining to the embodiment of 
the present invention; 

Fig. 5 is a flow chart showing the basic outline of 
the processing performed by the image processing 
apparatus pertaining to the embodiment of the 
present invention; 

Fig. 6 is a drawing to explain the operation principle 
cf the image processing apparatus pertaining to the 
embodiment of the present invention; 

Fig. 7 is a drawing to explain the operation principle 
of the image processing apparatus pertaining to the 
embodiment of the present invention; 

Fig. 8 is an external view of the image processing 
apparatus pertaining to the embodiment of the 
present invention; 
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Fig. 9 is a drawing to explain the operation principle 
of the image processing apparatus pertaining to the 
embodiment of the present invention; 

Fig. 10 is a drawing to explain the operation princi- 
ple of the image processing apparatus pertaining to 
the embodiment of the present invention; 

Fig. 11 is a drawing to explain the operation princi- 
ple of another image processing apparatus pertain- 
ing to the embodiment of the present invention; 

Fig. 12 is a summary block diagram of the image 
processing apparatus pertaining to the embodiment 
of the present invention; 

Fig. 13 is a flow chart showing the basic sequence 
to decide the camera orientations of the image 
processing apparatus pertaining to the embodiment 
of the present invention; 

Fig. 14 is a flow chart showing an outline of the 
match propagation sequence carried out by the im- 
age processing apparatus pertaining to the embod- 
iment of the present invention; and 

Fig. 1 5 is a drawing to explain the principle of mor- 
phing. 

DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0009] An embodiment of the present invention will be 
explained. 

[0010] First, the concept of the present invention will 
be explained. 

[0011] When a consumer accesses an Internet-based 
'total beauty site' that is implemented by an embodiment 
of the present invention, data regarding the number of 
product click-throughs, etc. made by the consumer is 
obtained and analyzed, and a database is generated. 
[0012] 3D face model data (which includes not only 
the face but the overall body data) of the consumer is 
obtained by the apparatus located in the shop, and is 
stored in the server of the total beauty site'. A prescribed 
fee is paid by the shop to the operator of the site to cover 
such charges as a fee for use of the technology of the 
apparatus and the data, a franchise fee, the sales mar- 
gin, and a consulting fee. 

[0013] The site operator provides to manufacturers, 
magazine publishers and the like (1) consumer prefer- 
ence information derived from the number of cllck- 
throughs and (2) additional Information classified by 
age, etc. Conversely, manufacturers, magazine publish- 
ers and the like pay the site operator consulting fees and 
data fees. 

[0014] Next, a basic outline of the system pertaining 
to an embodiment of the present invention will be ex- 



plained with reference to Figs. 1 through 4. 
[0015] In Fig. 1, connected to the Internet 11 are a 
server 1 0 to operate the lota! beauty site', a shop-based 
client 12 to obtain 3D face model data, a shop-based 
5 client 13 to perform simulation, a client 14 by which a 
consumer can carry out a simulation at the consumer's 
home, and an Internet cellular telephone 15 having a 
camera 15a to obtain a 3D face model. A computer 16 
belonging to a manufacturer, magazine publisher, etc. 
io may be connected to the server 10. Furthermore, the 
shop-based clients 12 and 13 may be connected to the 
server 10 through a method other than the Internet 11 . 
[0016] In Fig. 1, the consumer obtains her own 3D 
face model data in a beauty shop using the 3D image 
15 capturing apparatus 12a and the 3D image processing 
apparatus 12b connected to the shop-based client 12. 
The specific procedure followed will be discussed be- 
low. The shop-based client 12 sends the obtained 3D 
face model data to the server 10. The server 10 stores 
20 the received data. Once the data is stored in the server, 
the consumer can access the total beauty site' on the 
server 10 and carry out a makeup simulation from the 
shop-based client 1 2, the shop-based client 1 3 that has 
no image capturing apparatus, the home-based person- 
25 al computer 14 or the Internet cellular phone 15. The 
details of the makeup simulation will be described be- 
low. The behavior of the consumer while she is access- 
ing the total beauty site' is analyzed by the data control 
unit 10c and accumulated in a database. Because this 
30 data comprises information important in understanding 
the consumer's preferences, it is provided by the server 
10 to manufacturers, magazine publishers, etc. 16. 
Where the consumer has a camera-equipped computer 
or Internet cellular phone, a plurality of images obtained 
35 therefrom may sent to the server, enabling the server 
10 to constructs 3D face model data. In the above ex- 
planation, the makeup simulation is carried out by the 
server 10. Because the simulation processing compris- 
es advanced processing Including three-dimensional 
40 processing, by having it carried out by a server 10 ca- 
pable of advanced processing, the burden on the con- 
sumer-side personal computer may be reduced. It is ac- 
ceptable if the makeup simulation is carried out by the 
shop-based client 12. 
45 [001 7] Fig. 2 shows the basic construction of the serv- 
er 10. The member registration unit 10a shown in Fig. 
1 includes the member registration unit 100 and mem- 
ber database 101 shown in Fig. 2. The makeup simula- 
tion unit 1 0b shown in Fig, 1 includes the 3D face model 
50 data receiving unit 1 02, the 3D face model database 1 03 
and the makeup simulation providing unit 104 shown in 
Fig. 2. The data control unit 1 0c shown in Fig. 1 includes 
the user information analyzer 105, the control database 
106, the information processing unit 107 and the trans- 
55 mitting/receiving unit 108 shown in Fig. 2. 

[001 8] When makeup simulation is carried out, mem- 
ber registration must first be performed. A member reg- 
istration request is sent by the shop-based client 12 or 
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the home-based computer 14 to the server 10. When 
the member registration request is received, the mem- 
ber registration unit 1 00 writes member information into 
the member database 101 . 

[0019] The 3D model face data sent by the shop- 
based client 1 2 is received by the receiving unit 1 02 and 
stored in the 3D face model database 1 03. When a sim- 
ulation request is sent by the shop-based client 1 2 or 1 3 
or the home-based computer 14 to the server 10, the 
makeup simulation providing unit 104 determines 
whether or not the request is from a member, and if the 
request is from a member, it analyzes the contents of 
the request, reads out the member data from the data- 
base 103, carries out simulation in accordance with the 
request, and provides the simulation to the requesting 
party. 

[0020] At the same time, the actions taken by the con- 
sumer while at the total beauty site', for example, the 
contents of the simulation, clicks on specific products, 
clicks on banner ads, etc., are analyzed by the user in- 
formation analyzer 105 and are organized and stored in 
the control database 1 06 as consumer preference infor- 
mation. When an information supply request is received 
by the transmitting/receiving unit 108 from a manufac- 
turer, etc., the information processing unit 1 07 reads out 
prescribed data from the control database 1 06, subjects 
it to processing in accordance with the contents of the 
request, and then sends it to the requesting source. The 
details of the operation of the user information analyzer 
105 and the information processing unit 107 are ex- 
plained below. 

[0021] Fig. 3 shows the construction of the shop- 
based client 12. The shop-based client 12 includes a 
plurality of cameras 1a, 1b, etc., a 3D face model gen- 
erating unit 2, a makeup simulation unit 3, a display unit 
4, a touch panel 4a located on the display unit, a data- 
base 5 that stores 3D face model data, a pointing device 
6 such as a mouse, and a communication means 7 that 
connects to the server 1 0 or the Internet 1 1 . The 3D face 
model generating unit 2 includes a corresponding point 
search unit 2a, a three-dimensional shape recognition 
unit 2b and a geometric calculation unit 2c. Detailed ac- 
tions of these units will be described later. 
[0022] Fig. 4 shows the display apparatus 4. The 
three-dimensional image of the consumer is displayed 
in the area indicated by 4b, and color or pattern palette 
is shown in the touch panel 4a. Because a three-dimen- 
sional image is displayed in the display apparatus 4, a 
realistic makeup simulation may be experienced. For 
example, any type of makeup may be made through a 
one-touch operation of the touch panel 4a. A makeup 
style corresponding to various types of situations may 
be prepared in advance, such as party makeup, work 
makeup, etc., and the consumer's face may be repro- 
duced with the new makeup style based on a single 
touch of the touch panel. Alternatively, a makeup simu- 
lation may be executed manually. Once 3D face model 
data is sent to the server 10, these simulations may be 



earned out on a personal computer at home. 
Makeup simulation 

5 [0023] The contents of a simulation carried out using 
the makeup simulation providing unit 104 and the make- 
up simulation unit 3 will now be explained. 
[0024] Using this simulation, simulations of makeup, 
cosmetic surgery, clothing, perfume, accessories, hair 

io style, etc., may be provided based on 3D information. 
In addition, using morphing technology as described be- 
low, information to enable one to resemble one's favorite 
model may be obtained. For example, intermediate im- 
ages resembling a cross between oneself and one's fa- 

15 vorite model may be created through morphing technol- 
ogy, and the desired image may be selected. The user 
can learn what percentage of the image comprises her 
own features and what percentage comprises the mod- 
el's features. Simulation of not only one's face (the head 

20 area) but also one's entire body is possible as well. 
[0025] Several specific examples of such a simulation 
will now be explained. 

[0026] A simulation in which the level of beauty and 
degree of aging of the face are assessed and face iden- 

25 tification is carried out will first be explained. 

[0027] By analyzing the condition of ones facial skin 
and the light and dark areas reflecting the protrusions 
and indentations thereon, i.e., the state of the light areas 
and dark areas, the degree of one's beauty and appar- 

30 ent age can be objectively evaluated, identification of 
individual faces can be made, and the facial expression 
that best indicates and quantifies the person's emotional 
state can be objectively evaluated to a significant extent. 
[0028] Each individual's face has its own particular 

35 light and dark areas and areas comprising a mixture 
thereof. This means that each individual's face may be 
recognized based on these light, dark and mixed areas. 
Furthermore, depending on how one observes the face, 
one can observe changes in the shape of a person's 

<o light and dark areas that occur with changes in a per- 
son's facial expression. In other words, one's facial ex- 
pression changes with the contraction and relaxation of 
facial muscles, and such changes entail changes in the 
indentations and protrusions on one's face, i.e., in the 

45 light and dark areas of the face. Therefore, by observing 
these changes, even such an imprecise concept as a 
person's 'expression' can be quantified and objectively 
evaluated. 

[0029] Accordingly, an evaluation facial image, which 
50 comprises a facial image that has undergone light/dark 
processing, is used. To create such an image, first, an 
image of the subjects face must be captured and a facial 
image obtained. Next, through image enhancement 
processing of this facial image, particularly image en- 
55 hancement processing regarding the brightness of the 
image, an evaluation facial image comprising a plurality 
of areas having different levels of brightness is created. 
When this is done, the face may be evaluated for various 



4 



1SOOQD: cEP 11 3*701 A2J_> 



7 



EP 1 134 701 A2 



8 



purposes, such as the degree of beauty or of aging in 
the face, based on the contours of the light and dark 
areas in the evaluation image, or on the borders be- 
tween these areas. 

[0030] Furthermore, by comparing the evaluation fa- s 
cial image before and after face lift surgery, a determi- 
nation of the degree of aging may be carried out 
[0031] Using the above processes, a simulation for 
plastic surgery or makeup styles may be performed. 
[0032] Next, the process by which to correct the facial JO 
image will be explained. First, a desired face is chosen, 
images of a plurality of corrected candidate faces having 
varying degrees of resemblance to the desired face are 
created by alternating the original facial image using im- 
age processing such that the original facial image re- is 
sembles the desired face image, and a corrected facial 
image is obtained by selecting from among these a plu- 
rality of corrected candidate facial images. 
[0033] A model face may be used to choose the de- 
sired face. For the model face, a favorite television per- 20 
sonality or actress may be used. 
[0034] First, the desired face is selected. Where a 
makeup instructor is providing guidance regarding 
makeup application to a person wishing to wear make- 
up, for example, the desired face is chosen through the 25 
makeup instructor asking the prospective makeup wear- 
er about her preferences. The desired face may be cho- 
sen using a model face. For the model face, the pro- 
spective makeup wearer can use the face of a favorite 
television personality or actress. 30 
[0035] When the desired face is chosen, virtual make- 
up faces based on the desired face, i.e., images of vir- 
tual faces having the desired makeup style, are created. 
Through image processing such as the fusing of the de- 
sired face with the image of the face of the prospective 35 
makeup wearer, the faces of the prospective makeup 
wearer and of the desired face may be combined, bring- 
ing the prospective makeup wearer's face closer in ap- 
pearance to that of the desired face. From these virtual 
makeup faces, the ideal makeup face that is most de- 40 
sired by the prospective makeup wearer is determined. 
Specifically, because images of a plurality of virtual 
makeup faces exhibiting varying degrees of fusion or re- 
semblance between the prospective makeup wearer's 
face and the desired face are obtained through the *s 
above image processing, the preferred face can be cho- 
sen from among these faces as a desired virtual makeup 
face within the range of resemblance levels that may be 
obtained through the application of makeup. In this way, 
the ideal makeup face that is anticipated to be ultimately so 
obtained may be provided beforehand. In other words, 
the prospective makeup wearer can learn the final 
made-up look in a short amount of time. By selecting the 
desired face forming the basis for the makeup and seek- 
ing the ideal makeup face to obtain in connection with ss 
this face in this way, the final made-up look may be dis- 
played in a short amount of time. 
[0036] Once the ideal makeup face is chosen through 



the above makeup simulation process, a makeup tech- 
nique is deduced from the ideal makeup face. In other 
words, a series of makeup pointers by which to obtain 
the desired look, such as the areas where the eyebrows 
should be plucked or darkened, lines and areas where 
eye liner and eye shadow should be applied, eye shad- 
ow colors, areas where lipstick should be applied, and 
techniques for the application of foundation, are deter- 
mined based on a preset makeup program. Makeup is 
then applied to the prospective makeup wearer's face 
based on these makeup pointers. As a result, the ideal 
makeup face, i.e., the look that was accepted before- 
hand by the prospective makeup wearer, may be accu- 
rately reproduced on the face of the prospective makeup 
wearer. Put another way, any makeup desired by the 
prospective makeup wearer can be applied on her face 
in a short period of time. 

[0037] As described above, the makeup method of 
the present invention is characterized in that an ideal 
face based on a desired face, i.e., a model face, is cre- 
ated through image processing, and an important as- 
pect of this method is that the current face of the pro- 
spective makeup wearer and the model face are incre- 
mentally combined and brought closer together through 
image processing. 

[0038] Through this process, advanced corrections to 
the facial image may be made easily and in a short pe- 
riod of time. Furthermore, any desired makeup may be 
applied on the prospective makeup wearer in a short 
amount of time, and makeup possibilities based on a 
wide variety of cosmetic products may be effectively uti- 
lized. 

[0039] Next, the simulation of makeup facial images 
in beauty parlors, cosmetics shops, beauty schools, etc. 
will be explained. 

[0040] Makeup simulation drawing software uses a 
method in which the face as a whole is made up by ap- 
plying makeup to individual parts of the face. In this 
method, the sought makeup style is pasted on. For ex- 
ample, regarding eyebrows, a method is used in which 
a given eyebrow shape is chosen and pasted onto the 
existing eyebrow after it is matched to the size of the 
eyebrow on the facial image. Similarly, where lipstick is 
applied to the lips, a method is used in which a pre-ex- 
isting form is pasted on to an image. 
[0041] For the model makeup, images of the eye- 
brows, lipstick on the lips, powder (including foundation, 
eye shadow and blush) on the skin, and colored contact 
lenses are drawn . The image drawing operation for each 
facial component Is explained below. 

• Eyebrows: The eyebrow area is defined, the eye- 
brow in the original eyebrow area is shaved off, and 
the colorof the surrounding skin is drawn in. An eye- 
brow shape is chosen, and that shape is drawn in 
the eyebrow area. When this is done, processing is 
performed on a pixel-by-pixel basis in the eyebrow 
area, and the eyebrow is drawn in accordance with 
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a defined calculation formula. 

• Lipstick: The lip area onto which lipstick is to be ap- 
plied is defined, and the chosen lipstick color is ap- 
plied to the lip area. Here, image processing is car- 
ried out using the three elements of the hue, bright- 
ness and saturation. An image of the lip area is 
drawn by replacing the lip with the hue of the lipstick 
color, and converting the brightness and saturation 
of the original lips to the brightness and saturation 
of the lipstick. When this is done, operations such 
as glossing are also performed. Furthermore, the 
areas around the border between the lips and the 
skin are drawn such that the border between the 
lips and skin is a continuous color. 

• Powdering of the skin: An image is drawn in which 
the skin color value and the powder color value are 
mixed according to a specified ratio. Here, powder- 
ing includes the application of makeup such as 
foundation, eye shadow and blush. 

• Colored contact lenses: After the positions at which 
colored contact lenses are to be placed in the image 
(the positions at which the colored parts of the con- 
tacts are to be drawn) are defined, the color values 
of the colored contact lenses and of the iris are 
mixed according to a defined ratio in the display. 

[0042] The following variables should be recorded as 
makeup information for the model on whom makeup is 
applied using the above methods: 

• Eyebrows - eyebrow shape, eyebrow color (color 
value), position and size relative to the face 

• Lipstick - lipstick color value, degree of gloss 

• Powder - powder color value and density of appli- 
cation on a pixel-by-pixel basis 

• Colored contact lenses - color value of colored con- 
tact tenses 

• Facial image definition points 

[0043] The processing to apply the makeup of the se- 
lected model to the facial image of the user is carried 
out according to the following procedure. First, in a pre- 
liminary step, the facial image of the user is loaded into 
a computer using a digital camera, etc., and the -facial 
image is defined to match the image set for the model. 
Afterward, the same attribute values used for the mod- 
el's makeup are loaded in and applied to the defined 
facial image of the user. While the makeup is different 
for each facial component, the same materials are used 
for the user's eyebrows, lipstick and colored contact 
lenses that were used for the model, and images are 



drawn in the user's facial image using the same methods 
that were used for the drawing of the model face. How- 
ever, regarding powdering, because the density and the 
type of powder may differ depending on the area of the 
face, the correspondences of the respective pixels of the 
facial images of the model and the user are obtained, 
and makeup is applied to the facial image of the user. 
Specifically, using morphing technology, the corre- 
spondence of the respective pixels of the model facial 
image and the user's facial image is calculated, and the 
same powder that was present in a given pixel of the 
model image is used for the pixel of the user's skin hav- 
ing the same skin attribute at the same pixel of the model 
image. 

[0044] When a desired model facial image is selected 
from among the plurality of model facial images dis- 
played on the screen (where the makeup varies even 
though the same model is used, the different images are 
displayed as a separate menu item), the eyebrows, lip- 
stick, and colored contact lenses are automatically ap- 
plied using the methods explained with regard to the 
model makeup. However, because the powder applica- 
tion can differ on a pixel-by-pixel basis, powder is ap- 
plied using morphing technology. Two techniques are 
used in morphing; warp and dissolve. The first involves 
a method by which, when changing from one shape to 
another, the corresponding points are sought and the 
original shape is transformed, while the second involves 
transformation of the image by mixing the pre-change 
color and the post-change color in accordance with a 
defined ratio. The image drawing carried out for pow- 
dering in the present invention uses warp. 
[0045] It is often difficult to determine precisely what 
type of makeup one desires. Therefore, it is useful to 
have a model makeup style to refer to. When several 
models are registered beforehand, the makeup style 
used by the model can be applied to the user's facial 
image simply through the selection of a particular model, 
and therefore makeup styles that the user likes, or mod- 
el makeup styles that might be applied to one's face, 
can be selected, and many different makeup styles may 
be 'tried on'. 

[0046] Not only can the makeup style that was applied 
to the model facial image be transferred to the user's 
facial image, but because makeup is applied while pre- 
serving the user's facial contours and skin features, dif- 
ferent impressions may be created even with the same 
makeup style. For example, powder applied to a pale- 
skinned model reflects light differently than powder ap- 
plied to a user's suntanned skin. Consequently, the best 
makeup style for one's own face can be sought through 
simulation on a screen. 

[0047] Once the user's facial image data is loaded into 
a computer via an image data input apparatus and the 
facial area definition is made, simulation may be per- 
formed by changing the model an unlimited number of 
times. Furthermore, the skin color, the condition of the 
lips, or the quality of the face itself can change between 
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a facial image taken in summer and a facial image taken 
in winter. In the present invention, because makeup is 
applied while the characteristics of the original facial im- 
age are preserved, and these characteristics may 
change from time to time as described above, the effect 
of the makeup at different times may be checked and 
confirmed even if the same makeup is applied (i.e., the 
same model is selected) to the face of the same person. 
For example, because the method of the present inven- 
tion by which lipstick is drawn on the lips preserves the 
lines and shading of the lips even after the application 
of lipstick, one can clearly see the difference in the effect 
of the lipstick between when it is applied on rough lips 
during winter and when it is applied on fresh lips during 
summer. In the case of skin in particular, because skin 
color differs in summer and winter, the effect of makeup 
varies depending on the type of powder used. Using the 
automatic makeup simulation of the present invention, 
the differences in the effect of makeup based on the con- 
dition of the user's face may be directly confirmed on a 
screen as described above. Therefore, if facial images 
taken in the four different seasons are used, the best 
makeup style may be found in a short amount of time 
by applying the makeup styles of various different mod- 
els on one's facial image for the current season. 

User information analyzer 

[0048] Next, the user information analyzer 105 and 
the user processing unit 1 07 will be explained. These 
units carry out the following processes: 

(1) Statistical compilation of the user's Web usage 
information (number of click-throughs), etc.) 

(2) Analysis of the Web usage information, analysis 
of not just purchasing information but preference in- 
formation 

(3) Aggregation of purchasing information, supply 
of product preference information in new form 

(4) Analysis cross-referenced by age and region in- 
formation 

[0049] The user information analyzer 1 05 extracts, or- 
ganizes and supplies data by which to understand the 
overall user information based on the contents of the 
member database 101. It performs classification of ail 
of the registered users, and outputs basic user charac- 
teristic data such as the total number of registered us- 
ers, the ratio of men to women, the distribution of users 
by age and location of residence, etc. From the user be- 
havior history, which includes information on the degree 
of cooperation of each user with questionnaire surveys 
and on the frequency with which the user purchased 
products through the Internet home page, the class of 
users that would be best selected as target users may 
be learned. 

[0050] If the attributes of the target user class are 
clearly established, an effective business may be devel- 



oped by matching with the preferences of the target user 
class such basic elements as the contents of the total 
beauty site*, the style of writing, and the merchandise 
offered. Furthermore, problems may become more 

5 clearly defined. An example of such a problem might be 
that although women were originally targeted, there are 
fewer female registered users than expected. In such a 
case, responsive actions such as the posting of banner 
ads in information portals or sites accessed by a large 

10 number of women may be taken. 

[0051] It is also possible to prepare a number of dif- 
ferent brochures that are custom-tailored to the at- 
tributes of each group of users, and to send brochures 
with different contents to each group. One such bro- 

15 chure might focus on information regarding the most 
popular products among the merchandise handled by 
the total beauty site'. A better response would be antic- 
ipated in such a case that would be expected when the 
same brochure with the same content is sent to all mem- 

20 bers on a global basis. ^ 
[0052] The user information analyzer 105 performs 
access analysis. 'Access analysis* is the most basic 
analysis that measures the number of people that visit 
a particular site. If a site is equated to a shop, this 

25 number is eq urvalent to the number of customers visiting 
the shop. Analysis from various viewpoints may be car- 
ried out. For example, trends may be obtained regarding 
the number of customers visiting on each day of the 
week or during each time period of the day, the number 

30 of customers who enter but leave without purchasing, 
and the number of customers visiting each area of the 
site. 

[0053] Access analysis is performed using the three 
indices of number of hits, PV (page view), and number 

35 of visitors, 

[0054] The number of hits is a value that indicates the 
number of 'data sets' that were requested to be sent 
from a particular site. The unit of measurement for 'data 
sets' here is the number of data files in a computer. If 

40 the data set is a home page and the home page includes 
a large amount of graphic data, the number of hits in- 
creases accordingly. Conversely, even if a large amount 
of information is contained in one page, if that data con- 
sists of one text file, it is counted as T hit. 

45 [0055] A more practical index is PV (page view). It in- 
dicates the total number of Internet home pages viewed 
in connection with a particular site. While this index en- 
tails the shortcoming that any single home page counts 
as 1 PV regardless of the amount of information con- 

50 tained therein, it is a standard index used to measure 
the value of a medium or the effect of an ad, such as a 
banner ad, that is displayed on a one-page basis. 
[0056] There are cases in which the number of PVs 
associated with the top page of a particular site is 

55 deemed the number of visitors. Because PV indicates 
the number of total viewed pages, the number of differ- 
ent people that have viewed the page cannot be ob- 
tained. The number of visitors is an index for solving this 
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problem. Naturally, where one person accesses the top 
page repeatedly, each access is counted, and therefore, 
the number of visitors in this case is only an approximate 
number. 

[0057] In order to measure the number of visitors 
more precisely, such methods as a 'cookie' or a 'regis- 
tration system' must be used. 
[005B] A cookie not only enables behavior analysis, 
but is also effective for one-to-one marketing. The use 
of a cookie allows the behavior of a particular person (or 
more accurately, the behavior of a Web browser) within 
the site to be tracked. 

[0059] For example, suppose it is learned that con- 
sumers who request a lipstick simulation during a make- 
up simulation session are significantly more likely to re- 
quest lipstick brochures than consumers who do not re- 
quest a lipstick simulation. 

[0060] If this trend is utilized property, the target pop- 
ulation may be approached more effectively. If a bro- 
chure request page is forcibly shown to users who re- 
quest a lipstick simulation, the rate of brochure requests 
may be increased substantially. 
[0061] Through the use of a cookie, information may 
be provided in a customized fashion that matches each 
user's behavior and preferences. In order to implement 
this feature, the site must have cookie issuance and da- 
tabase functions. 

[0062] While personalization based on the use of a 
cookie cannot completely specify each individual, a reg- 
istration system can overcome this shortcoming. 
[0063] The address, telephone number, e-mail ad- 
dress and name are registered beforehand, and an ID 
and password used exclusively by the total beauty site' 
are issued. A member accessing a site enters a mem- 
ber-only page when she inputs her ID and password. 
[0064] By having the users log in, the identity of each 
user, the pages they visit, and their behavior while 
logged in can be tracked by the site. At the same time, 
a page dedicated to the user may be displayed after 
login. 

[0065] If the areas of information desired by a user 
are obtained through responses to a questionnaire dis- 
tributed at the time of registration, news that matches 
the user's stated interests may be posted on a particular 
page. 

[0066] From not only the registration information, but 
also from behavior information that indicates the areas 
of the site most commonly visited by the user, the indi- 
vidual's preferences may be derived and information 
matching these preferences may be displayed. 

Three-dimensional face model generating unit 

[0067] The three-dimensional face model generating 
unit 2 will now be explained. 

[0068] One known method of image processing is the 
morphing technique. Morphing is a computer graphics 
(CG) technology developed in Hollywood, U.S.A. Ac- 



cording to this method, two different images are used, 
for example, images of the faces of two persons, and 
one of the images is gradually changed on the screen 
to the other image, thereby providing a series of images 

5 showing such change. Using the morphing technology, 
it is possible to create a series of images in which, for 
example, a white tiger turns into a young woman. 
[0069] When two images A and B are given, the mor- 
phing process is roughly as follows. First, the corre- 

10 sponding feature points between image A and image B 
are obtained (e.g., eye and eye, nose and nose). This 
process is normally performed by an operator. When the 
correspondences are found, feature point p of image A 
is gradually changed in a time-consuming process to 

is feature point q of image B, resulting in the image series 
as described above, 

[0070] In CG, an image is generally made of a large 
number of triangular elements. Therefore, morphing is 
performed by changing the triangle of feature point p in 
20 image A to the triangle of feature point q in image B while 
maintaining the correspondence between them. This 
will be described further with reference to Fig. 1 5. In this 
figure, triangle A is part of image A, and triangle B is part 
of image B. The apexes p1 , p2, p3 of triangle A each 
23 correspond to apexes q1 , q2 and q3 of triangle B. In or- 
der to convert triangle A to triangle B, the differences 
between p1 and q1 , p2 and q2, and p3 and q3 are cal- 
culated, and then respectively added to each of the 
apexes p1 , p2, p3 of triangle A. By adding all (100%) of 
30 these differences, triangle A is converted to triangle B. 
It is also possible to add portions of these differences 
instead of the whole differences, e.g., 30% or 60% 
thereof, in such case, the intermediate figures between 
triangle A and triangle B can be obtained. For example, 
35 in Fig. 1 5, triangle A' is a model example of an addition 
of 30% of the difference, and triangle B' is a model ex- 
ample of an addition of 60% of the difference. 
[0071] Fig. 5 is a flowchart showing an outline of the 
processing of the apparatus/method according to an 
to embodiment of the present invention. The image data 
(signals) obtained from a plurality of cameras 1a, 1b, in 
Fig.3. are input into a front view image generating unit 
2. In front view image generating unit 2, a corresponding 
point searching unit 2a searches the mutually corre- 
45 sponding points by analyzing the plurality of images. 
These corresponding points are analyzed by a three- 
dimensional shape identifying unit 2b, and the three-di- 
mensional shape of the object is identified. Based on 
the identified results, the viewing rays are set, and the 
so data is geometrically converted or varied based on the 
set viewing rays, thereby generating a front view image 
that would be gained by looking into a minor. Each of 
the above-mentioned processes will be described in fur- 
ther detail below. Furthermore, camera 1 need only be 
55 a plurality of cameras, regardless of whether 2, 3, 4 or 
more. Two or three are desirable from the practical as- 
pect. 

[0072] The processing of the front view image gener- 
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ating unit will be described in further detail based on 
Figs. 6 and 10. Fig. 6 is a model view of a digital mirror 
comprising cameras 1 at the left and right upper ends 
and the lower center of a plate-shaped liquid crystal dis- 
play apparatus (LCD) 4. An object 100 is placed on the 
normal vector intersecting substantially the center of 
LCD 4. Normally, the face of the user is located at this 
position, but for convenience of explanation, a quadran- 
gular pyramid is used as an example. When quadran- 
gular pyramid 100 is shot by cameras 1a, 1b, and 1c, 
images 1 00a, 1 00b, and 1 00c are obtained. Image 1 00a 
is shot by camera la, and viewed from LCD 4, this image 
is a view of pyramid 100 from the left side. Image 1 00b 
is shot by camera 1 b, and is a view of pyramid 1 00 from 
the right side. Image 100c is shot by camera 1c, and is 
a view of pyramid 100 from the bottom. If there are at 
least two images seen from different viewpoints located 
relatively adjacent to each other, then it is possible to 
identify a unique three-dimensional shape from a plural- 
ity of two-dimensional images through a geometrical 
calculation processing similar to the stereoscopic view 
processing. In order to perform this processing by a 
computer, it is necessary to specify the feature points. 
In the present example, the apexes of quadrangular pyr- 
amid 1 00 are selected. When the feature points have 
been specified for all images, the correspondence be- 
tween these feature points is calculated. In this way, it 
is analyzed at which position in each image the same 
portion of pyramid 100 is located. Based on this analy- 
sis, the three-dimensional shape of pyramid 1 00 is iden- 
tified. According to image 100a, the apex is on the left 
side, so it is clear that pyramid 1 00 is at the left of cam- 
eral la. In this way, the three-dimensional shape is iden- 
tified. Thereafter, the viewpoint is set for example sub- 
stantially in the center of LCD 4, and based on this view- 
point, an image of pyramid 1 00 is generated. For exam- 
ple, image 100 as shown in Fig. 7 is obtained. 
[0073] In Fig. 3, signal processing unit 3 receives the 
front view image processed as above from front view 
image generating unit 2, and performs various process- 
ing such as displaying the object or a reflection of the 
object such as gained by conventional mirror reflection, 
etc. Examples are the zoom and wide angle processes. 
A certain portion of the whole image reflected in a mirror 
is instantaneously enlarged or reduced. The selection 
of the portion to be enlarged or reduced and the 
processing to be performed is designated by a pointing 
device 6 such as a mouse. If the surface of LCD 4 is a 
touch panel, it is possible to touch an arbitrary portion 
of the image to enlarge or reduce such portion instan- 
taneously. 

[0074] Fig. 8 is a variation of the apparatus in Fig. 3. 
Three CCD cameras 1 a, 1 b and 1 c are provided around 
LCD 4. At the back of LCD 4, a computer is provided 
which functions as front view image generating unit 2 
and signal processing unit 3. These are all stored in one 
case. 

[0075] Now, the whole processing of the apparatus/ 



method according to an embodiment of the present in- 
vention will be described in outline. According to the 
flowchart in Fig. 5, two or more images A, B,... from two 
or more different viewpoints are obtained (S1). 

5 [0076] Next, the correspondence between feature 
points in image A and image B is calculated (S2). Fea- 
ture points may be edges, comers, texture, etc. 
[0077] The difference between corresponding feature 
points in image A and image B is calculated (S3). 

w Through this processing, the extraction of the necessary 
features points and the difference between them 
(amount of change) can be gained as required for the 
morphing process. 

[0078] The present embodiment is also a drawing ap- 

15 paratus and method for performing morphing of images 
of three-dimensional objects. In order to draw images of 
three-dimensional objects, the position of the object 
within a space must be determined, and, according to 
the present drawing apparatus and method, it is possi- 

20 ble to draw images of three-dimensional objects without 
directly requiring the three-dimensional position. 
[0079] The movement principle will be described by 
using Figs. 9 and 10. As shown in Figs. 9(a) and (b), a 
cone 201 and a cube 202 are arranged within a certain 

25 space and shot by two cameras la and 1 b. As the view- 
points of cameras 1 a, 1 b differ, the obtained images are 
also different The images obtained by cameras la, 1b 
are as shown in Figs. 10(a) and (b). Comparing these 
two images, it is clear that the positions of cone 201 and 

30 cube 202 are different. Assuming that the amount of 
change in the relative position of cone 201 is y, and that 
of cube 202 is x, then Fig. 1 0 shows that x<y. This is due 
to the distance between the object and the cameras. If 
the values of x and y are large, the feature points are 

35 near the camera. On the other hand, if such values are 
small, the feature points are far from the camera. In this 
way, the distances between the object and the cameras 
are clear from the differences between corresponding 
feature points in the different images. Utilizing this char- 

40 acteristic, the feature points are sorted according to the 
differences (S4), and the images are written in order 
from that with the smallest difference (meaning the im- 
age shot by the camera farthest to the object) to the larg- 
est difference (S5). Portions near the camera are over- 

45 written and displayed, but portions far from the camera 
(hidden portions) are deleted through the overwriting. In 
this way, it is possible to adequately reproduce an image 
in three-dimensional space without using depth informa- 
tion. 

so [0080] The apparatus shown in Figs. 5 to 8 is able to 
display an image shown from a different viewpoint than 
camera 1 by processing the image obtained from cam- 
era 1 as shown in Fig. 5. For example, It is possible to 
use the images of a face from the right, from the left and 

55 the bottom to generate and display the image of the face 
seen from the front. Also, by applying morphing 
processing to the face seen from the right and the left, 
it is possible to display the face from various angles, as 
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if the camera viewpoint had continuously moved. The 
apparatus in Figs. 5 to 8 can be used quasi as a digital 
form mirror (hereinafter the "digital mirror*). 
[0081] It is also possible to use the apparatus in Figs. 
3 to 5 as a digital window simulating an actual window. 
By displaying various scenes on a liquid crystal televi- 
sion, the present invention provides a display apparatus 
for the window to be used in substitution of the actual 
window. Conventional display apparatuses merely dis- 
played images, e.g. scenery, seen from a fixed view- 
point, being unable to express small changes in scenery 
occurring from changes in the viewpoint position at the 
actual window. By utilizing the apparatus or method ac- 
cording to the present embodiment, it is possible to rec- 
ognize the position of the person, i.e. the positron of the 
viewpoint, so by changing the display according to the 
viewpoint position, an even more real scenery display 
is possible. For example, Fig. 11 shows a liquid crystal 
apparatus ("digital window") W and a person standing 
before it. in Fig. 11(a), a cone and cube are arranged 
within a virtual space, and this situation is displayed on 
liquid crystal apparatus W. If the person is at position b, 
the image shown in Fig. 11 (b) will be displayed on liquid 
crystal apparatus w, and if the person is at position c, 
then the image shown in Fig. 11(c) will be displayed. In 
this way, by displaying an adequate screen according 
to the viewpoint position, the user will feel as if he were 
turning his head at an actual window. 
[0082] The digital mirror and digital window process- 
ing methods above are common in that they include a 
processing of determining the position of an object with- 
in a three-dimensional space by calculating the corre- 
spondence of feature points between a plurality of im- 
ages. In the digital mirror, the position measurement 
precision is desirably high, as the measurement preci- 
sion of the three-dimensional position directly affects the 
image precision. However, in the digital window, there 
is no large feeling of strangeness even if the viewpoint 
position is somewhat inaccurate. Therefore, the digital 
window does not require as high a measurement preci- 
sion of the position as the digital mirror. A processing 
apparatus/method for the digital mirror will be hereinaf- 
ter referred to as the facial image generator and a 
processing apparatus/method for the digital window the 
scenery image generator. Both will be now described in 
further detail. 

[0083] The facial image generator conducts its 
processing using three cameras and a trifocal tensor 
suited as constraint. The scenery generator conducts 
its processing using two cameras and the epipolar ge- 
ometry as constraint. Conventionally, it was difficult to 
find correspondences only by comparing the three im- 
ages of the three cameras, but by using the space con- 
straints of the three cameras, the correspondence 
search can be performed automatically. 



Facial Image Generator 

[0084] An example of the processing of three images 
with different viewpoints from three cameras will be de- 
5 scribed below. 

1. Feature point detection unit 

[0085] Three images with different viewpoints are in- 
fo put into three feature point detection units 10a to 10c. 
Feature point detection units 1 0a to 1 0c outputs a fist of 
feature points also called points of interest. If the object 
has a geometrical shape such as triangles or squares, 
the apexes thereof are the features points. In normal 
15 photograph images, points of interest are naturally good 
candidates forfeature points as points of interest are by 
their very definition image points that have the highest 
textureness. 



[0086] Correlation units lla and lib and a robust match- 
ing unit 12 make a seed finding unit. This unit functions 
to find an aggregate of Initial trinocular matches (con- 

25 straint of the positions of three cameras) that are highly 
reliable. Three lists of points of interest are input into this 
unit, and the unit outputs a list of trinocular matches of 
the points of interest called seed matches. Correlation 
units lla and 11b establish a list of tentative trinocular 

30 matches. Robust matching unit finalizes a list of reliable 
seed matches using robust methods applied to three 
view geometric constraints. 

2.1 Correlation unit 

35 The movements of correlation units lla and 1 1 b 

will be described below. These units perform the 
processing of three lists of points of interest in three 
images output from feature point detection unit 1 0a 
to 10c. The 2NCC (zero-mean normalized cross- 
4° correlation) correlation measure is used for finding 
correspondences. By using the ZNCC correlation 
measure, it is possible to find the correspondence 
between images even if the size of the object is 
somewhat different between such images or the im- 
45 ages are somewhat deformed. Therefore, the 
ZNCC correlation is used for matching seeds. 

The ZNCCx(A) at point x = (x,y) T with the shift 
A = (A x , A Y ) T is defined to be: 

50 L,(/(x + i) - 7(x))(/'(x + A + i) - 7'(x + A)) 

(I,(/(x + i) -7(x)) 2 L,(/*(x + A + i) - 7'(x + A)) 2 ) 1/2 

where l'(x) and H(x) are the means of pixel lumi- 
55 nances for the given window centered at x. 

2.2 Robust matching unit 

Next, the binocular matches from correlation 
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unit 11 are merged into one single trinocular match 
by robust matching unit 12. Robust matching unit 
12 receives input of a list of potential trinocular 
matches from correlation unit 11 and outputs a list 
of highly reliable seed trinocular matches. A robust 
statistics method based on random sampling of 4 
trinocular matches in three images is used to esti- 
mate the 12 components of the three-view con- 
straints to remove the outliers of trinocular matches. 
When the same object is shot by three cameras and 
three images from different viewpoints are gained, 
the same point in the object in each of the three im- 
ages (e.g., position of feature point) can be uniquely 
defined from the position of the object, the camera 
position and the camera direction according to cer- 
tain rules. Therefore, by determining whether the 
points of interest in the list of trinocular matches 
gained from correlation unit 11 satisfies such rules, 
it is possible to obtain the list of points of interest of 
the correct trinocular matches. 

Given u = (u,v), u' = (u'y) and u° = (u",v") the 
normalized relative coordinated of the trinocular 
matches, the three-view constraints are completely 
determined by the following 1 2 components to t 12 : 

f 4 u + t Q v+t u t/ + (q u* = 0, 



t 2 u+t 6 v+ + f 10 u" = 0, 



f 3 u+ f 7 v+ / 12 </ + f 9 v" = 0, 



^ i/+f 5 v+t^v 0 * f 10 v" = 0. 



3 Unit of auto-determination of camera orientations 

[0087] Now, a camera orientation auto-determination 
unit 1 3 will be described below. The classical off-line cal- 
ibration of the whole system is hardly applicable here 
even though 3 cameras may be a priori fixed, but their 
orientations could be still variable. Therefore, camera 
orientation auto-determination unit 13 determines the 
camera orientation in order to constrain the match prop- 
agation. In other words, camera orientation auto-deter- 
mination unit 13 receives input of a list of seed matches 
from robust matching unit 12 and outputs the orientation 
of the camera system. 

[0088] Now, the basic ideas of camera orientation au- 
to-determination unit 1 3 will be described below. At first, 
th e th ree-view constra ints tj , .... t, 2 are optimally re-com- 
puted by using all trinocular inlier matches. The extrac- 
tion of camera orientations directly from the three-view 
constraints for later usage is based on an original ob- 
servation that the problem of affine cameras is convert- 
ed into a nice problem of 1 D projective cameras. 



[0089] For those skilled in the art, it is evident that an 
elegant 10 projective camera model first introduced in 
L. Quan and T. Kanade "Affine structure from line cor- 
respondences with uncalibrated affine cameras" IEEE 

5 Transactions on Pattern Analysis and Machine Intelli- 
gence, 1 9(8): 834-845, August 1 997 occurs on the plane 
at infinity for the usual affine cameras. All directional 
quantities are embedded on the plane at infinity, there- 
fore encoded by the 1 D projective camera. The 1 D cam- 

10 era is entirely governed by its trifocal tensor T IJk (provid- 
ing a strong constraint) such that T^u'uV* = 0. 
[0090] From the above aspects, the procedure of de- 
termining the camera orientations according to the 
present embodiment is as follows. 

15 

S11: Convert 2D affine cameras into 1D projective 
cameras 

Using tensor-vector mapping defined by 4(a - 1) + 
2(b - 1) + c 1 between the tensor components and 
& the three-view constraint components converts the 
triplet of affine cameras represented by ti into the 
triplet of 1 D cameras represented by T^. 
S1 2: Extraction of epipoles 
The 1 D camera epipoles can be extracted from the 
25 tensor by solving, for instance, IT jk e 2 l = 0 for the 
epipoles e 2 and e 3 in the first image. The other 
epipoles can be similarly obtained by factorizing the 
matrix \ k e\ for e^ and e' 3 and T.^e^ for e^ and 
e° 2 . 

S13: Determination of camera matrices M* = (H, h) 
and M' = (H\ h') and the camera centers c, c* and c" 
It is first straightforward that h = e', and h' = e", . The 
homographic parts of the camera matrices are de- 
termined from T fjk = H,ih* - hlH',* Then, the camera 
centers and the 2D projective reconstruction can be 
determined from the camera matrices as their ker- 
nels. 

S14: Update of the projective structure 
The known aspect ratio for the affine camera is 
equivalent to the knowledge of the circular points 
on the affine image plane. The dual of the absolute 
conic on the plane at infinity could be determined 
by observing that the viewing rays of the circular 
points of each affine image plane are tangent to the 
absolute conic through the camera center. 
S15: Determination of camera orientation parame- 
ters 

Transforming the absolute conic to its canonical po- 
sition therefore converts all projective quantities in- 
to their true Euclidean counterparts. Euclidean 
camera centers give the orientation of the affine 
cameras and the affine epipolar geometry is de- 
duced from the epipoles. 
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4. Constraint match propagation unit 

[0091] Now, a constraint match propagation unit 14 
for expecting a maximum number of matches in three 



srmnrv <fp 113-4701*2 t > 



11 



21 



EP1 134 701 A2 



22 



images will be described below. This unit 14 receives 
input of a list of seed matches and camera orientation 
parameters from camera orientation auto-determination 
unit 13 and outputs dense matching in three images. 
[0092] After obtaining the initial seed matches, it 
comes the central idea of match propagation from the 
initial seed matches. The idea is similar to the classic 
region growing method for image segmentation based 
on the pixel homogeneity. The present embodiment 
adopts region growing to match growing. Instead of us- 
ing the homogeneity property, a similarity measure 
based on the correlation score is used. This propagation 
strategy could also be justified as the seed matches are 
the points of interest that are the local maxima of the 
textureness, so the matches could be extended to its 
neighbors which have still strong textureness though not 
a local maxima. 

[0093] Ail initial seed matches are starting points of 
concurrent propagations. At each step, a match (a, A) 
with the best ZNCC score is removed from the current 
set of seed matches (S21 in Fig. 1 4). Then new matches 
are searched in its 'match neighborhood 1 and all new 
matches are simultaneously added to the current set of 
seeds and to the set of accepted matehes-under con- 
struction (S22). The neighbors pixels a and A are taken 
to be all pixeis within the 5 x window centered at a and 
A to ensure the continuity constraint of the matching re- 
sults. For each neighboring pixel in the first image, we 
construct a list of tentative match candidates consisting 
of all pixels of a 3 x 3 window in the neighborhood of its 
corresponding location in the second image. Thus the 
displacement gradient limit should not exceed 1 pixel. 
This propagation procedure is carried out simultaneous- 
ly from the first to the second and the first to the third 
image, and the propagation is constrained by the cam- 
era orientation between each pair of images. Only those 
that satisfy the geometric constraints of the camera sys- 
tem are propagated. Further, these two concurrent prop- 
agations are constrained by the three- view geometry of 
the camera system. Only those that satisfy the three- 
view geometry of the camera system are retained. 
[0094] The unicity constraint of the matching and the 
termination of the process are guaranteed by choosing 
only new matches not yet accepted. Since the search 
space is reduced for each pixel, small 5x5 windows are 
used for ZNCC, therefore minor geometric changes are 
allowed. 

[0095] It can be noticed that the risk of bad propaga- 
tion is greatly diminished by the best first strategy over 
all matched seed points. Although seed selection step 
seems very similar to many existing methods for match- 
ing points of interest using correlation, the crucial differ- 
ence is that propagation needs only to take the most 
reliable ones rather than taking a maximum of them. 
This makes our algorithm much less vulnerable to the 
presence of bad seeds in the initial matches. In some 
extreme cases, only one good match of points of interest 
is sufficient to provoke an avalanche of the whole tex- 



tured images. 

5. Re-sampling unit 

5 [0096] Now, a re-sampling unit 15 will be described 
below. The dense matching may still be corrupted and 
irregular, re-sampling unit 15 will regularize the match- 
ing map and also provide a more efficient representation 
of images for further processing. Re-sampling unit 15 

10 receives input of the dense matching in three images 
from constraint match propagation unit 14 and outputs 
a list of re-sampled trinocular matches. 
[0097] The first image is initially subdivided into 
square patches by a regular grid of two different scales 

is 8x8 and 16 x 16. For each square patch, we obtain ail 
matched points of the square from the dense matching. 
A plane homography H is tentatively fitted to these 
matched points u, < — > u', of the square to look for po- 
tential planar patches. A homography in P 2 is a projec- 

20 tive transformation between projective planes, it is rep- 
resented by a homogeneous 3x3 non singular matrix 
such that X,u', = Hu,, where u and u* are represented in 
homogeneous coordinates. Because a textured patch 
is rarely a perfect planar facet except for manufactured 

25 objects, the putative homography for a patch cannot be 
estimated by standard least squares estimators. Robust 
methods have to be adopted, which provide a reliable 
estimate of the homography even if some of the 
matched points of the square patch are not actually lying 

30 on the common plane on which the majority lies. If the 
consensus for the homography reaches 75%, the 
square patch is considered as planar. The delimitation 
of the corresponding planar patch in the second and the 
third image is defined by mapping the four comers of 

35 the square patch in the first image with the estimated 
homography H. Thus, a corresponding planar patches 
in three images is obtained. 

[0098] This process of fitting the square patch to a 
homography is first repeated for all square patches of 
<o the first image from the larger to the smaller scale, it 
turns out all matched planar patches at the end. 

6 Three-view joint triangulation unit 

45 [0099] Now, a three-view joint triangulation unit 16 will 
be described below. The image interpolation relies ex- 
clusively on image content without any depth informa- 
tion and is sensitive to visibility changes and occlusions. 
The three view joint triangulation is designed essentially 

so for handling the visibility issue. Three-view joint triangu- 
lation unit 16 receives input of the re-sampled trinocular 
matches and outputs joint three-view triangulation. The 
triangulation in each image will be Delaunay because of 
its minimal roughness properties. The Delaunay trian- 

55 gulation will be necessarily constrained as we want to 
separate the matched regions from the unmatched 
ones. The boundaries of the connected components of 
the matched planar patches of the image must appear 
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in ail images, and therefore are the constraints for each 
Deiaunay triangulation. 

[0100] The joint three-view triangulation is defined as 
fulfilling the following conditions. 

- There is one-to-one vertex correspondence in three 
images. 

- The constraint edges are the boundary edge of the 
connected components of the matched regions in 
three images. 

There is one-to-one constraint edge correspond- 
ence in three images. 

In each image, the triangulation is a constraint De- 
iaunay triangulation by the constraint edges. 

[0101] A natural choice to implement this joint three- 
view triangulation is a greedy-type algorithm. 



10 



claims, and such variations are included within the 
scope of the present invention. 
[0106] As used herein, means is not limited to physi- 
cal means but includes cases where the functions of 
such means are realized through software. Further- 
more, the functions of one means may be realized 
through two or more physical means, and the functions 
of two or more means may be realized through one 
physical means. 



15 



7 View interpolation unit 

[0102] Now, a view interpolation unit 17 will be de- 
scribed below. According to view interpolation unit 17, 
any number of in-between new images can be generat- 
ed, for example, images seen from positions between a 
first and a second camera. These in-between images 
can be generated from the original three images. View 
interpolation unit 1 7 receives input of the three-view joint 
triangulation results and outputs any in-between image 
l(ct, p, y) parameterized by a, 0, and ysuch that 

[0103] The view interpolation processing is performed 
according to the following procedures. 

1 . The position of the resulting triangle is first inter- 
polated from three images. 

2. Each individual triangle is warped into the new 
position and a distortion weight is also assigned to 
the warped triangle. 

3. Each whole image is warped from its triangula- 
tion. In the absence of depth information, a warping 
order for each triangle is deduced from its maximum 
disparity to expect that any pixels that map to the 
same location in the generated image are arriving 
in back to front order as in the Patter's method. All 
unmatched triangles are assigned the smallest dis- 
parity so that they are always warped before any 
matched triangles. 

4. The final pixel color is obtained by bleeding three 
weighted warped images. 
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Claims 

1 . A three-dimensional beauty simulation client-server 
system comprising: 

a shop-based client that obtains and transmits 
three-dimensional shape data regarding a us- 
er; and 

a server that comprises a makeup, simulation 
unit that receives and stores said three-dimen- 
sional shape data from said shop-based client 
and carries out makeup simulation based on 
said three-dimensional shape data in response 
to the user's requests, and a data control unit 
that analyzes the user's operation record and 
generates administrative information. 



2. 
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[01 04] Furthermore, the similar idea developed for fa- 
cial image generation from 3 images could be extended 
to either 2 or N images with reasonable modification of 
the processing units. Other objects than face images 
could also be processed in a very similar manner. 55 
[0105] Needless to say, the present invention is not 
limited to the embodiment described above and may be 
varied within the scope of the invention described in the 



so 5. 



6. 



The three-dimensional beauty simulation client- 
server system according to claim 1 , further compris- 
ing a client that can access said server, wherein 
said server provides a makeup simulation in re- 
sponse to requests from said client. 

The three-dimensional beauty simulation client- 
server system according to claim 1 , further compris- 
ing a cellulartelephonethat has a data transmission 
function and can access said server, wherein said 
server provides a makeup simulation in response 
to requests from said cellular telephone. 

The three-dimensional beauty simulation client- 
server system according to claim 1, wherein said 
server further comprises a member registration unit 
that stores member registration information, and 
wherein said server provides makeup simulations 
to users registered beforehand in said member reg- 
istration unit. 

The three-dimensional beauty simulation client- 
server system according to claim 1, wherein said 
server transmits the operation record and/or admin- 
istrative information regarding said users via a com- 
puter network. 

The three-dimensional beauty simulation client- 
server system according to claim 1 , wherein said 
shop-based client comprises: 
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a plurality of cameras to obtain images of the 
user as seen from a plurality of viewpoints; 
a corresponding point search unit that receives 
each item of image data obtained from the plu- 
rality of cameras, analyzes the plurality of im- 5 
ages, and searches for corresponding points 
that correspond to each other; 
a three-dimensional shape recognition unit that 
analyzes the searched corresponding points 
and recognizes the three-dimensional shape of '0 
the target object; 

a geometric calculation unit that sets a line of 
sight based on the recognition results from said 
three-dimensional shape recognition unit, and 
generates an image from a prescribed line of 
sight through geometric conversion of the data 
based on the set line of sight; 
a display unit that displays the image generated 
by said geometric calculation unit; and 
communication means thattransmits the image 20 
data generated by said geometric calculation 
unit to said server. 

7. The three-dimensional beauty simulation client- 
server system according to claim 6, wherein said 25 
corresponding point search unit and said geometric 
calculation unit comprises: 

a feature point extraction unit that extracts fea- 
ture points from each of said plurality of images; 30 
a correlation calculating unit that seeks corre- 
lation among the feature points of said plurality 
of images and seeks combinations of said fea- 
ture points; 

a matching unit that discards combinations 35 
having a low feasibility from among said com- 
binations of feature points based on the condi- 
tion that the images were seen from said plu- 
rality of viewpoints; 

a camera orientations determining unit that *o 
seeks the positions of said plurality of view- 
points and the directions of the lines of sight; 
and 

a match propagation unit that, under the condi- 
tions imposed by the positions of said plurality 
of viewpoints and the direction of said lines of 
sight obtained by said camera orientations de- 
termining unit, selects combinations of feature 
points starting with those having superior geo- 
metric and statistical reliability and adjusts the so 
analysis range of the images of said target ob- 
ject. 

8. The three-dimensional beauty simulation client- 
server system according to claim 6, wherein said ss 
corresponding point search unit and said geometric 
calculation unit comprises: 



a feature point extraction unit that extracts fea- 
ture points from each of said plurality of images; 
a correlation calculating unit that seeks corre- 
lation among the feature points of said plurality 
of images and seeks combinations of said fea- 
ture points; 

a matching unit that discards combinations 
having a low feasibility from among said com- 
binations of feature points based on the condi- 
tion that the images were seen from said plu- 
rality of viewpoints; 

a camera orientations determining unit that 
seeks the positions of said plurality of view- 
points and the directions of the lines of sight; 
a match propagation unit that, under the condi- 
tions imposed by the positions of said plurality 
of viewpoints and the direction of said lines of 
sight obtained by said camera orientations de- 
termining unit, selects combinations of feature 
points starting with those having superior geo- 
metric and statistical reliability and adjusts the 
analysis range of the images of said target ob- 
ject; 

a resampling unit that normalizes the matching 
map obtained by said match propagation unit; 
a three-dimensional position measurement unit 
that determines the position of said target ob- 
ject in a three-dimensional space based on the 
normalized matching map; and 
a view interpolation unit that generates images 
seen from viewpoints different from said plural- 
ity of viewpoints based on the determined 
three-dimensional position of said target ob- 
ject. 

9. The three-dimensional beauty simulation client- 
server system according to claim 6, wherein said 
corresponding point search unit and said geometric 
calculation unit comprises: 

a feature point extraction unit that extracts fea- 
ture points from each of said plurality of images; 
a correlation calculating unit that seeks corre- 
lation among the feature points of said plurality 
of images and seeks combinations of said fea- 
ture points; 

a matching unit that discards combinations 
having a low feasibility from among said com- 
binations of feature points based on the condi- 
tion that the images were seen from said plu- 
rality of viewpoints; and 
a match propagation unit that, under the geo- 
metric constraints imposed by the lines of sight, 
selects combinations of feature points starting 
with those having superior geometric and sta- 
tistical reliability and adjusts the analysis range 
of the images of said target object. 
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10. The three-dimensional beauty simulation client- 
server system according to claim 6, wherein said 
corresponding point search unit and said geometric 
calculation unit comprises: 

a feature point extraction unit that extracts fea- 
ture points from each of said plurality of images; 
a correlation calculating unit that seeks corre- 
lation among the feature points of said plurality 
of images and seeks combinations of said fea- 
ture points; 

a matching unit that discards combinations 
having a low feasibility from among said com- 
binations of feature points based on the condi- 
tion that the images were seen from said plu- 
rality of viewpoints; 

a match propagation unit that, under the geo- 
metric constraints imposed by the lines of sight, 
selects combinations of feature points starting 
with those having superior geometric and sta- 
tistical reliability and adjusts the analysis range 
of the images of said target object; 
a resampling unit that normalizes the matching 
map obtained by said match propagation unit; 
a three-dimensional position measurement unit 
that determines the position of said target ob- 
ject in a three-dimensional space based on the 
normalized matching map; and 
a view interpolation unit that generates images 
seen from viewpoints different from said plural- 
ity of viewpoints based on the determined 
three-dimensional position of said target ob- 
ject. 



1 1 . A three-dimensional beauty simulation server com- 
prising a makeup simulation unit that receives and 
stores three-dimensional shape data of a user from 
a shop-based client and carries out makeup simu- 
lation based on the three-dimensional shape data 
in response to requests from the user, and a data 
control unit that analyzes the operation record for 
said user and generates administrative information, 

wherein said makeup simulation unit comprises 
a receiving unit that receives said three-dimen- 
sional shape data, a database that stores the 
received three-dimensional shape data, and a 
makeup simulation providing unit that provides 
a makeup simulation in response to requests 
for such simulation; and 
wherein said data control unit of said server 
comprises a user Information analyzer that re- 
ceives the operation history of the user and an- 
alyzes the trends therein, a control database 
that stores the analyzed data, an information 
processing unit that reads out data from the 
control database in response to external re- 
quests and processes the data in accordance 



with said requests, and a transmitting/receiving 
unit that transmits the output of said information 
processing unit to the requesting source and re- 
ceives requests from the requesting source. 

5 

12. The three-dimensional beauty simulation server ac- 
cording to claim 11 , wherein said makeup simula- 
tion providing unit analyzes the condition of the us- 
er's facial skin and the light and dark areas that in- 
fo dicate the protrusions and indentations thereon, 

and evaluates the user's facial expression based on 
the results of such analysis. 

1 3. The three-dimensional beauty simulation server ac- 
cording to claim 11, wherein said makeup simula- 
tion providing unit obtains a facial image of the user, 
displays for the user a plurality of target facial im- 
ages stored beforehand for allowing the user to se- 
lect one of these images, combines said user facial 
image and said target facial image in a plurality of 
predetermined ratios, and supplies a plurality of 
combined facial images to the user. 

1 4. The three-dimensional beauty simulation server ac- 
cording to claim 11, wherein said makeup simula- 
tion providing unit supplies facial images seen from 
freely chosen viewpoints. 

15. A system comprising means for deriving a 3-D im- 
age of a user, means storing information regarding 
user preferences regarding certain predetermined 
criteria, and means for generating an image of the 
user modified in accordance with said stored infor- 
mation. 

16. A method of producing a modified image of a user, 
the method comprising determining information re- 
lating to user preferences regarding certain prede- 
termined criteria, deriving a 3-D image of a user, 
and modifying the image of the user in accordance 
with said information. 

17. A method as claimed in claim 1 6 wherein the user's 
behaviour is monitored to determine said informa- 
tion relating to user preferences. 



15 



20 



25 



30 



35 



40 



45 



50 



55 



Jsnona<EP u 34701 A2 i > 



15 



EP1 134 701 A2 



F IG. 1 



MANUFACTURERS. 
MAGAZINE PUBLISHERS. 
ETC 



SERVER 



! MEMBER 
REG ' STrAT I QH 
UNIT 



; MAKEUP 

SIMULATION 
! UNIT 



DATA 
CONTROL 
UNIT 



15 

L_ 

I CELLULAR TELEPHONE HAVING 
! DATA COMMUNICATION FUNCTION 
|( INTERNET CELLULAR TELEPHONE) 



Xj7-iOP-BASEO 
j CLIENT 




15a 

i 

_i — 



CAMERA 



OPERATEO TO PERFORM 
SIMULATION 



CLIENT 
(HOME-BASEO 
COMPUTER) 



SHOP -BASED 
CLIENT 



CPERATEO TO PERFORM 
SIMULATION 



OPERATED TO PERFORM 
"SIMULATION 



123- 



12; 



3D IMAGE 

PROCESSING 

APPARATUS 



30 IMAGE 

CAPTURING 

APPARATUS 



USER 



16 

SOOaO: <EP 11 34701 A2J_> 



EP1 134 701 A2 



101 

Jl 



MEM3SR 08 






F 1 G. 2 






100 

1 






MEM8ER 

REGISTRATION 
UNIT 










30 FACE 
MOOEL 08 



106 

V 



CONTROL 08 



MEMBER REGISTRATION 
REQUEST 



MAKEUP SIMULATION 
PROVIDING UNIT 



105 



SIMULATION SUPPLY 
REQUEST 

•PROVIDE SIMULATION 



102 



30 FACE MOOEL 
OATA RECEIVING 
UNIT 



30 FACE 
MOOEL OATA 



USER INFORMATION 
ANALYZER 



107 



108 



INFORMATION ^ 
PROCESSING UNIT 



TRANSMITTING/RECEIVING 
UNIT 



COOat); <S> 1 1 J4701 A2 J_> 



17 



EP 1 134701 A2 




^5 



7\ 



SOOOa <EP_1134701A2J_> 



18 



EP1 134 701 A2 



F IG.4 



4 

4b ( 4a 




19 

«nOOft«6P U3470IA2 I > 



EP1 134 701 A2 



F I G. 5 



OBTAIN TWO OR MORE IMAGES 
A. B- . . SEEN FROM TWO OR 
MORE DIFFERENT VIEWPOINTS 


S1 




t 




SEEK CORRESPONDENCE OF 
FEATURE POINTS AMONG 
IMAGES A.B. ETC. 


^S2 


< 


i 




SEEK AMOUNT OF MOVEMENT OF 
CORRESPONDING FEATURE POINTS 
AMONG IMAGES A.B. ETC. 






i 




SORTING 8ASE0 ON AMOUNT 
OF MOVEMENT 


^— S * 








IMAGE ORAW BEGINNING WITH 
FEATURE POINTS HAVING 
SMALLEST AMOUNT OF MCVEM6NT 


^ ,S5 



20 

iSOOQD: <EP 11 34701 A2J_> 



EP1 134 701 A2 



F IG 6 



1b 




tSOOQD: <£P 1 1 34701 A2_l_> 



21 



EP 1 134 701 A2 




22 

ISDOCIO: <EP 11 34701 A2J_> 



IV 



EP1 134 701 A2 



F I G. 8 




2.'3 



&T>mi> <FP. 1134701A2 I > 



23 



EP1 134 701 A2 



F I G. 9 

la 

1 



(5) 



'< 201 -V / 

' V 


V 


202 

/ 




\\\ 


V 




0 




24 

ISOOCID: <EP 1 1 34701 A2J_> 



EP1 134 701 A2 



FIG. 10 



(3) 




snorao- <ep ii3470iA2 i > 



25 



EP1 134 701 A2 




SDOQD: <£P 11 34701 A2J_> 



26 



EP1 134 701 A2 



F I G. 12 



FIRST IMAGE 



10a 



FEATURE POINT 
OETECTOR 



SECONO IMAGE 

10b 
t £ 



FEATURE POINT 
OETECTOR 



THIRO IMAGE 



10c 

i d 



FEATURE POINT 
OETECTOR 



1ia 



CORRELATION 
UNIT 



CORRELATION 
UNIT 



ROBUST MATCHING UNIT 



AUTOMATIC CAMERA ORIENTATIONS 
DETERMINING UNIT 



SEEO DISCOVERY UNIT 
~12 

J 3 



MATCH PROPAGATION UNIT FOR 
PROPAGATING MATCHES FOUND 
UNOER GEOMETRIC LIMITATION 



14 



RESAMPLING UNIT 



THREE-VIEW JOINT 
TRt ANGULATION UNIT 



VIEW INTERPOLATION UNIT 



,15 



16 



J7 



27 



EP 1 134 701 A2 



FIG. 13 



r 



tf'V/fftT 20 AFFIHE CAMERA 
:C PROJECTION CAMERA 



TO 



EXTRACT EPIPCLSS ' 




CgTERMlNE CAMERA MATRIX 
AND CAMERA CENTER 



S13 



i 



| UPCATE PROJECTION STRUCTURE 



S14 



DETERMINE CAMERA 
CAVITATIONS PARAMETERS 



S15 



28 

ISOOCID: <6P 1 134701 A2 I > 



I 



EP1 134 701 A2 



F IG. 14 



DISCARD INITIAL SEED 
MATCHES HAVING BEST 
ZNCC SCORES 


S21 








OISCGVER NEW MATCHES 
IN 'MATCH PROXIMITY'. 
ADD TO MATCH GROUP 


S22 



tannnrv h*47T)1A2 i > 



29 



EP1 134 701 A2 



F l G.15 




30 



THIS PAGE BLANK (uspto) 



